Checkpoint IPS

Mailman List Mirror (Read Only)
21

I've never heard a plausible anecdote, much less seen meaningful

statistics,

of these devices actually 'preventing' anything.

People tend to hear what they want to hear. Surely your claim can't be that
an IPS has never, in the history of Earth, prevented an attack or exploit.
So it's unclear to me what you're actually trying to say here.

And the fact that well-known evasion techniques still work against these
devices today, coupled with the undeniable proliferation of compromised
hosts residing within networks supposedly 'protected' by these devices,
militates against your proposition.

Your tendency of making blanket statements is somewhat baffling given the
multitude of intricacies, details, and varying circumstances involved in a
complex topic like this. To me, it's indicative of an overly-simplified
and/or biased way of looking at things.

In any case, go ahead and stick with your router ACLs and (stateful!)
proxies. Different strokes.

-Terry

22

it's not evil; it's stupid because enforcing symmetry creates a potentially
unnatural stress in a network which will revert to asymmetry, given half a
chance.

Nick

23

Firewalls aren't much help except at that extreme edge, anyhow.

24

Count up the number of *actual* attacks they have stopped that wouldn't
have been stopped otherwise, and contrast it to the number of times they've
been used as the *basis* for an attack (DDoS via state exhaustion, for starters)
or their failure has caused operational issues. Remember that one of the
three security pillars is "Availability".

Still think they're a good idea?

25

> But there's no overstating the usefulness of a properly-tuned IPS for
> attack prevention

I've never heard a plausible anecdote, much less seen meaningful

statistics,

of these devices actually 'preventing' anything.

I think it depends upon where you put them, and whether or not you have
skilled people involved. Given good placement, and experienced management,
I have seen the usefulness of these devices ... by personally reviewing the
logs and being the recipient of drop alerts of the devices in terms of what
they can reject.

I have, however, run into many, many situations in which these devices
demonstrably degraded the security posture of network operators,
particularly when placed in front of servers or broadband access networks.
For example, they're laughably easy to DDoS due to state exhaustion -

which

is what is the main point of the presentation you reference.

Sometimes we get in to corner cases too easily where the negative is easily
applied. Yes, they can be DDoS'd. Yes they can be useless in the hands of
the unskilled and unknowing.

On the other hand, given good placement in strategic places, and maintained
appropriately, they can live up the their expectations.

And the fact that well-known evasion techniques still work against these
devices today, coupled with the undeniable proliferation of compromised
hosts residing within networks supposedly 'protected' by these devices,
militates against your proposition.

Well.... again, yes, they may not get all zero-day exploits. That is
another corner case. But they can certainly prevent getting hit by the same
old stuff over and over again. I've seen drop rules for the Bash issue, and
the openssl issue put in place, and when implemented, prevent the further
spread of the malicious payloads. There must some sort of value in that?

26

No - patch the servers.

27

What if you are a hosting company and those aren't your servers to patch?
What about the time to patch 200+ servers versus configuring one location?
What if you have to schedule the staff and maintenance window to patch the servers?
What if you have legacy equipment that you must continue using, but the vendor is slow to provide the patch.

There is a huge difference in what is good network/security designs between content providers, transit networks, eyeball networks, corporate networks, universities, etc... One size doesn't fit all.

28

What if you are a hosting company and those aren't your servers to patch?

Then it isn't the operator's problem.

What about the time to patch 200+ servers versus configuring one location?

Operators should have sufficient automation to do this quickly. If not, they're Doing It Wrong.

What if you have to schedule the staff and maintenance window to patch the servers?

See above.

What if you have legacy equipment that you must continue using, but the vendor is slow to provide the patch.

There are other ways (reverse proxies, on-box systems like ModSecurity, et. al.); or take them offline.

29

You make so many assumptions, it completely negates any reasonable point you are trying to make:

There are other ways (reverse proxies, on-box systems like ModSecurity,
et. al.); or take them offline.

What if the box isn't Linux? What if it isn't a web server. What if proxies don't work well with the protocol the boxes uses. What if it's an appliance a business unit made you setup. There a thousands of permutations like that. Many times you don't get to make the correct choices, you have to work with what you have. Any IPS, statefull firewall, application level gateways, proxies, etc. have their places.

In a content provider network (facebook, etc...) only using stateless protection because of massive DDOS is a reasonable argument. But like I said, one size doesn't fit all, or in this case, many.

Like it's been said before, I strongly support my competitors following your advice.

30

Sorry - I've done the jobs, all of them. They can be done properly, and are done properly by clueful operators.

Oh, and what are operators who deploy these things supposed to do about *vulnerabilities in these devices themselves*? That's a huge problem, they present a juicy attack surface, and exploits are discovered regularly. That's in the presentation, as well.

I've heard these same tired arguments over and over again. Folks tend to change their tune when their entire production infrastructure is rendered unavailable by a tiny DDoS which could be sourced from a single smartphone; it's just sad that so many are only ready to listen and learn after they've suffered serious production-impacting outages.

If all it took to achieve *real* security - as opposed to 'compliance' or vendor marketing 'security' - were to write a check or cut a P.O. and drop some middlebox/middleblade in the network, we wouldn't be in the permanent state of security emergency in which we find ourselves.

*Real* security mostly consists of *doing things*. It requires skilled, experienced people who have both broad and deep expertise across the entire OSI model, are well-versed in architecture and the operational arts, and who understand all the implications of scale.

Unfortunately, such people are relatively rare, even within the self-selected ranks of network operators - as several posts on this thread clearly demonstrate.

31

Count up the number of *actual* attacks they have stopped
that wouldn't have been stopped otherwise

Many.

and contrast it
to the number of times they've been used as the *basis* for
an attack (DDoS via state exhaustion, for starters)

Zero, on my networks.

or their failure has caused operational issues.

Zero, on my networks. Unless "operation issues" means traffic fails over
without a hitch.

Still think they're a good idea?

Yep. And thanks for asking.

If you can't deploy IPS's in such a way that they don't make your network
less secure via DDoS susceptibility, or reduce availability due to
non-existent or subpar redundancy/survivability engineering, then you
shouldn't deploy IPS's.

-Terry

32

*Real* security mostly consists of *doing things*. It requires skilled,

experienced

people who have both broad and deep expertise across the entire OSI

model, are

well-versed in architecture and the operational arts, and who understand

all the

implications of scale.

And if there's one person qualified to comment on what "real security" is,
it's a person who has "never heard a plausible anecdote of [IPS] devices
actually 'preventing' anything." :slight_smile:

-Terry

33

That's right - especially if such a person spent a not-inconsiderable amount of time working for the world's largest manufacturer of such devices, and heard no such plausible anecdote during his time there, even though he asked repeatedly for same.

34

Zero, on my networks.

Which highlights the importance of broadness of experience, of knowledge and understanding of the experiences of others, and understanding of the implications of scale.

If you can't deploy IPS's in such a way that they don't make your network
less secure via DDoS susceptibility, or reduce availability due to
non-existent or subpar redundancy/survivability engineering, then you
shouldn't deploy IPS's.

By their very nature, it's impossible to do so.

35

Which highlights the importance of broadness of experience, of
knowledge and understanding of the experiences of others, and
understanding of the implications of scale.

It highlights the importance of knowing what you're doing in the real world,
on networks that exist and which you actually understand intimately,
end-to-end. And maybe also of not working for vendors, since these two
things are often mutually exclusive.

By their very nature, it's impossible to do so.

Nope. But in any case, I've violated what I said this morning about not
turning this into a big back-and-forth. (Sorry.)

-Terry

36

It highlights the importance of knowing what you're doing in the real world, on networks that exist and which you actually understand intimately,
end-to-end.

Absolutely. At least one of the parties in this discussion has such knowledge of and experience on real-world networks of considerable scale, and is not infrequently engaged hands-on in the preservation of availability on said networks in fraught circumstances.

And maybe also of not working for vendors, since these two things are often mutually exclusive.

Again, absolutely. At least one of the parties has seen firsthand the extremely negative impact of the devices in question on a broad array of large-scale production networks ever since said devices were first introduced in the 1990s, having been awakened at 0Dark30 on numerous occasions with pleas for assistance because 'the network is down', 'the data center is offline', 'our entire wireless broadband network is down', et. al., due to the manifest failings of such devices.

Anyways, enough. This topic has been thoroughly discussed multiple times on this list and on others of an operational nature; if you believe it wise to discount the well-understood negative impact of such devices on availability, that is your choice.

37

+100% agree.

...Skeeve

*Skeeve Stevens - Founder & Chief Network Architect*
eintellego Networks Pty Ltd
Email: skeeve@eintellegonetworks.com ; Web: eintellegonetworks.com

Phone: 1300 239 038 ; Cell +61 (0)414 753 383 ; Skype: skeeve

Facebook: eintellegonetworks <http://facebook.com/eintellegonetworks> ;
Twitter: eintellego <https://twitter.com/eintellego>

LinkedIn: /in/skeeve <http://linkedin.com/in/skeeve> ; Expert360: Profile
<https://expert360.com/profile/d54a9>

The Experts Who The Experts Call
Juniper - Cisco - Cumulus Linux - Cloud - Consulting - IPv4 Brokering

38

There's room for a good engineered strategy for protection which won't turn into a point of failure in the overall networking topology.

For decades, since first rainbow series books were written and military “strategy” started to be added to information security, it’s always been about defense in depth and layered defense. Today those buzzwords became an incredibly “bullshit bingo” on sales force strategy on selling magical boxes and people tend to forget the basics. Layers and the “depth” is not a theory just to be added to drawings and keynote presentations.

Considering a simplistic topology for 3-tier (4 if you count T0) depth protection strategy:

(Internet)--[Tier-0]--(Core Router)--[Tier1]--(core switch)--[Tier2]--(DMZ)--[Tier3]--(Golden Secret)

One security layer (tier, whatever) is there to try to fill the gap from the previous one.

How deep you have to dig depends on who you are. If you are the end organization willing to protect the golden secret, how complex is your topology, or if you are the carrier, the telecom the company worried only about BGP, PPS, BPS and availability other than the actual value for what's the real target for the attack (if not availability)

In summary, in my experience what will (not) work is:

1) Tier 0 & Tier 1
On border, core, (Tier0) or on Tier-1 protection layers (typical “firewall/dmz” topological position)

- Memory and CPU exaustion will shut down your operations (increase latency and decrease availability) easily, if you have a Protecting Inbound Proxy (Web Application Firewall, for the sales jargon), Stateful Firewall or IPS.

The thing here is, you are insane or naive if you believe a finite state machine with finite resources can protect you from a virtually infinite (unlimited) source of attacks. No matter how much RAM you have, how much CPU cycles you have, they are finite, and since those “amazing stateful w/ deep inspection, scrub, normalization and reassembly” features on a firewall will demand at least 4K RAM per session and a couple CPU cycles per test, you have a whole “line rate” (1.4Mpps / 14Mpps for 1GbE 10GbE ports) attack potential, and come on, if a single or a 3-way packets sequence will cost you a state, it’s an easy math count to find out you are in a bad position trying to “secure” those Tier0 and Tier1 topological locations. It's just easier and cheaper for the one attacking, than for your amazing firewall or IPS.

So what to do here? Try to get rid of most automated/scripted/simple attack you can. You can do ingress filtering, a lot of BCP, protection against SNMP/NTP/DNS amplification, verify reverse path, (verrevpath, antipsoof, verrevreachability), and many good / effective (but limited) protection with ACL, data plane protection mechanisms, BGP blackholing, Null Routing (sending stuff to disc0 on BSD, null0 on Cisco, etc) and Stateles Firewalling.

Also, an IDS sensor might fit here, because if CPU/Memory starvation happens on an IDS sensor, the worst thing will happen is some packets getting routed without proper processing. But still, they will get routed.

Always stateles, always simple tests. No Layer7 inspection of any form, no load balancer, WAF, whatever. No regex, hell no! No memory pointers that can point to dark processor/memory locations (again, no regex).

2) On Tier 2 protection (A defense depth that comes after core protection and after Tier-1):

- Will miserably fail if you use stateful firewall in excess
- Will fail even quicker if you use a WAF or IPS

Many “security engineers” and “security experts” (or worse, security tools vendors and developers) excessively use stateful tests on transport protocols that are simply… stateles.

What good you have on stateful firewalling… ICMP? UDP? DDP? IGMP? come on, all those state timers are worthless and your memory limits will reach very soon. Remember, in the average, 4K RAM at least, per stateful session. Much more resources are needed for IPS, much much more for inbound proxy (WAF), not to mention CPU.

Will you add an IPS here? What for, other than easing putting down and making your network and services unavailable?

Proxy? Mod Security? Hell no! Not here. Did you check how many regex and rules you have in the “base_rules” collection for a Top 10 OWASP protection on Mod Security? What about vendor specifics rules, or Sans Top 25 specifics.

This is just not the place.

Also, let’s rationale, what is your real benefit on deep inspection stateful filtering on those SSH sessions allowing for your trusted locations only? Really, what good will it make? Drop it stateles, allow it stateles! If you really believe stateful worths something for protocols such as SSH, do it somewhere else (Tier3 or host-based).

Focus on stateful protection only for what really needs some protection of this kind. Packets related, fragile services for packets arbitrarily assembled. Maybe SIP (SIP, not RTP), HTTPS, HTTP. Layer3/Layer4 fragile and public services, after some stateles inspection was already done on Tier1, should deserve stateful protection only. Not your whole network! Not your whole set of services or services.

And no ICMP, no UDP do deserve stateful.

Also, pick up a good tool for this.

Most sysadmins, security guys or network operators never notice how their tools may betray 'em hardly (by deault).

For example one use OpenBSD PF firewall, every rule you make will “automagically” become an stateful rule. And this is not good! This is terrible! This “auto” behavior makes the security engineer blind, since the rules he is writing is not the rules he is adding to kernel.

OpenBSD’s PF has a “no state” option and nobody uses it! It means you are doing it wrong… UDP/ICMP/TCP rules always have “keep state” added, if you don’t explicitly set “no state”. Beware!

The same is valid for Linux Netfilter, and 9 in 10 commercial firewalls (checkpoint, mikrotik, fortinet, whatever…).

FreeBSD’s ipfw on the other hand is stateles by default. It means if you don’t explicitly add “keep-state” there, it’s stateles. Which is good, unless you explicitly want a rule to be stateful. It’s excellent as a default behavior for protection layers not close to your "golden egg provider". On the other hand, ipfw by default has a limit of 4096 states which is TOO LOW. Beware too!

Regarding IPS/WAF/Proxy or “Next Generation” firewalls, this is not the place to add it.

On the other hand you need some level of extra protection firewalls will not provide, and some Layer7 inspection on Tier2 will be good.

But not Proxy! Not mod security! and not IPS! (no WAF)

IMHO, an IDS will fit good here.

IDS, not IPS, not IDP. Not a magical solution…

Why, from my past experiences, an IDS approach here will fit? Due it’s passive nature. If your IDS (say, Suricata, Bro, keeping on open source, or your commercial option of choice) starves on CPU or memory, what’s the worst thing to happen?

You will have a high number of PPS on that perimeter port passing without getting processed/inspected. Your overall security will decrease, but you still have Tier3 (and maybe other tiers) to fulfill the gap! Packets that still can be processed should have active response in place taking care of a lot of attacks.

What I mean is, if you have limited (and you do have limited) processing and memory power, say you can IDS inspect 400Kpps but a 1Mpps attack is ongoing, you have 40% inspection power, but packets not processed still get routed! Without latency and without packet loss because it’s an IDS and not an IPS. It’s not inline. It’s passive, sitting there. Limited resources, priority and lower kernel CPU scheduler relevance.

And for those 40% (very bad scenario I am drawing here) you still have active response in place, rerouting, reconfiguring stateles firewall, stateles ACLs and null routes on upper tiers (tier 0, tier 1, routers, switches) and lower tiers (tier 3 reconfigured upon tier 2 decision).

What you will do is try to fill the gap on next tiers, or increase your filtering capabilities that are still stateles or passive.

For many business, this is the end for added protection layers. A big ISP, transport provider, content delivery business... more protection from this point is something for the specific end business, and completely related to their needs, their business model, process, responsibilities and overall evaluated requirements. Not a general model to fit.

But for everyone else up to this tier you have relevant security mechanisms and tecnology added, and still low starvation/exaustion risks due to the stateles and passive nature of the approach.

3) On Tier 3, a protection strategy for your “golden eggs” provider, the golden secret, your business secret and intelligence

Usually, this protection level is for the corporate strategy. The business, not the carrier, the service provider or the network operator. And is a business specific requirement. Meaning it may not exist at all!

Now, for me, here is where you add more stateful inspection (still, only what is actually efficient, not that god damn echo reply/request wasting memory to be tracked down - useless!).

Here is a good place for a WAF, after firewall and IDS protection took place. WAF is a not a panaceia for anything, it's aimed for specific attacks against applications and protocols, is not resilient to coward attacks (volumetric, L3/L4 exaustion, etc). And a proxy in the end is always a web server, so inherits all it's fragility, and therefore it must be protected as well, before it can actually protect anything else.

Host Intrusion Detection central servers (receiving information gathered from HIDS on the actual hosts) also fits Tier3. As well as other host-based controls that may add telemetry information to a central location.

Talking about telemetry, it's important everywhere, and while generating flows or snmp info grabbing may impact processing usage on critical core devices, those extra added boxes should also passively help telemetry, with flow generation or minimum capability for snmp servings.

Nothing here is new. I am talking about, again, basic stuff discussed for decades on colored cover military books, drafts, discussions and BCPs and really old stuff discussed for people who may be already dead, sometimes (Itojun and other samurais' missed). I'm only mentioning the basic 3 tech domains (firewall, ids, proxy), but the other two basic (pen test, vuln scan) that are more process than technology are important as well.

For me, and again this is very personal opinion, I never run an IPS unless the customer "really wants to" (or a stupid compliance requirement really requires to). An IDS+Active Response is as good as IPS, and the only extra benefit an IPS will add compared to IDS is related to "single packet attacks", that ones that will pass quickly enough before the active response blocks it. But "single packet" attacks are related to poorly written software (or unpatched / not fixed software) since it's not really an attack, it's a trigger to bad code misbehavior and should be addressed on the host.

This is a very simple model, and easy to understand. However how many situations we've seen big companies getting completely unavailable or AAA getting broken because people insist to buy (and sell) "miracle boxes" added to core locations, and those miracle boxes will have amazing deep inspection firewalls or IPS or DLP (whatever it means, Data Loss, Data Leak, it means whatever you want to buy)...

There may be a place for those stuff, but it's not on core. Nor on second level protection layers.

In the end, ISO27002, PCI-DSS, CIA and AAA triads, what are they worth if you add an IPS to your core? When memory is exausted, processing is starved, you will have packet loss, latency, or you will be completely offline. And what's CIA if you "security features" are breaking Integrity due to missing packets, or breaking full Availability at all? What you have from CIA? Only confidentiality? Better take that plug off. Same for AAA, if Authorization and Authentication are broken or failed due to exaustion/starvation what you get? Accounting/Auditing? So you will sit and read the logs to find out what went wrong?

Discouraging firewall/IDS/proxy protection layers is as bad as over leveraging it.

Dosage is what distinguishes the poison from the vaccine.

39

An IPS doesn't have to be in line.

It can be something watching a tap and scripted to use something else
to block traffic (e.g. hardware filtering options on a router that can
handle it).

An IDS tied into an internal RTBH setup to leverage uRPF filtering in
hardware can be pretty effective at detecting and blocking the typical
UDP attacks out there before they reach systems that don't handle that
as gracefully (e.g. firewalls or host systems). Even if you keep it
from being automated and just have it be an IDS that you can have a
human respond to is pretty valuable.

40

Using flow telemetry for this scales much, much better. One could easily set something like this up using open source flow telemetry collection/analysis tools.

Of course, giving attackers the ability to spoof the IP addresses of their choice and then induce your network infrastructure into blocking said IP addresses isn't necessarily optimal, IMHO. I'm not a big fan of any kind of auto-mitigation for this reason - it's best to have a human operator in the loop.

If one is determined to do this kind of auto-mitigation, it's probably a good idea to whitelist certain things which ought never to be S/RTBHed via appropriate route filtering on the trigger and/or edge devices where traffic will be dropped.