we have been getting reports lately about unsecured UDP chargen servers
in our network being abused for reflection attacks with spoofed sources
In the UDP implementation of the protocol, the server sends a UDP
datagram containing a random number (between 0 and 512) of characters
every time it receives a datagram from the connecting host. Any data
received by the server is discarded.
We are seeing up to 1500 bytes of response though.
This seems to be something new. There aren't a lot of systems in our
network responding to chargen, but those that do have a 15x
amplification factor and generate more traffic than we have seen with
abused open resolvers.
Anyone else seeing that? Anyone who can think of a legitimate use of
chargen/udp these days? Fortunately I can't, so we're going to drop
19/udp at the border within the next hours.
We got hit with this in September. UDP/19 became our most busiest port overnight. Most of the systems participating were printers. We dropped it at the border, and had no complaints or ill effects.
> In the UDP implementation of the protocol, the server sends a UDP
> datagram containing a random number (between 0 and 512) of characters
> every time it receives a datagram from the connecting host. Any data
> received by the server is discarded.
We are seeing up to 1500 bytes of response though.
This seems to be something new. There aren't a lot of systems in our
network responding to chargen, but those that do have a 15x
amplification factor and generate more traffic than we have seen with
abused open resolvers.
Anyone else seeing that? Anyone who can think of a legitimate use of
chargen/udp these days? Fortunately I can't, so we're going to drop
19/udp at the border within the next hours.
*checks her calendar* I for a second worried I might have woken up from
a 20 year long dream....
Are these like machines time forgot or just really bag configuration
choices?
Not sure. The affected IPs are strongly clustered around the Faculty of
Medicine, so from experience I would assume stone-old boxes. But not
sure yet.
Hmmm. Do you not run a default deny at your border, which would catch this sort of thing? Granted thats not always possible I suppose. Maybe block all UDP you dont specifically need? Do you have an ids/ips? If not, look at SecurityOnion on a SPAN port, it will provide great insight into whats happening.
Generally these sort of legacy services are only used for malicious activity and will light up an ids/ips like a Christmas tree.
They must be old boxes. I cant think of any recent os distributions which would even have these services listening, let alone installed.
Dropping the TCP and UDP "small services" like echo (not ICMP echo), chargen and discard as part of default firewall / filter policies probably isn't a bad idea. Those services used to be enabled by default on Cisco routers, but that hasn't been since probably around 11.3 (mid-late 90s).
Other than providing another DDoS vector, I'm not aware of any legitimate reason to keep these services running and accessible. As always, YMMV.
While blocking it at your border is probably a fine way of mitigating the problem, I would recommend doing an internal nmap scan for such things, finding the systems that respond, and talking with their owners.
Please report back to NANOG after talking to them letting us know if the owners were still using SunOS 4.x boxes for some reason, had accidentally enabled chargen, or if some malware had set up the servers. Inquiring minds would like to know!
For a while, it was possible to spoof packets to create a TCP connection from a
machine's chargen port to its own discard port and walk away while it burned to
the ground. Fun times.
Other than providing another DDoS vector, I'm not aware of any legitimate
reason to keep these services running and accessible. As always, YMMV.
They are useful for troubleshooting and diagnostic purposes. Just be
sure to limit the maximum possible response rate and bandwidth for any
source network, and be sure to truncate the length of the response
to the length of the original query, so they cannot be used for
amplification. If you can't do that, then shut them off
The risk that they be used to DoS the server that runs those services remains.
All of the above plus very poorly managed network / network security. (sadly a Given(tm) for anything ending dot-e-d-u.) a) why are *printers* given public IPs? and b) why are internet hosts allowed to talk to them? I actually *very* surprised your printers are still functional if the whole internet can reach them.
Being an edu, even if they aren't globally reachable, there is *plenty* mischievousness already inside the borders! Securing a campus from the world... easy; securing a campus from it's own users... good luck with that.
Guess what, they have /16s, they use them, and they like
the ability to print from one side of campus to the other. Are you
suggesting gigantic NATs with 120,000 students and faculty behind them?
I have a hard time blaming a school for this. I have an easy
time wondering why printer manufacturers are including chargen support
in firmware.
Indeed I have. Which is why I haven't for a great many years. Academics tend to be, well, academic. That is, rather far out of touch with the realities of running / securing a network. I've used the work "incompotent" in previous conversations, but that's mostly a factor of overwork in an environment where few people are ever fired for such.
Guess what, they have /16s, they use them, and they like
the ability to print from one side of campus to the other. Are you
suggesting gigantic NATs with 120,000 students and faculty behind them?
Guess what, there are companies that have /8's, and they manage to keep their network(s) reasonably secured. I'm not talking about uber-large NAT; I'm talking about proper boundry security. If you cannot figure out how to keep the internet away from your printers, you should look into other lines of employment. Limiting access of the residential network into the departmental networks, is one of the first things in the design of a res-net. Otherwise, there's 25k potential script kiddies (or infected home computers now on your network) waiting to attack everything on campus. But we're headed into the weeds here...
I have a hard time blaming a school for this. I have an easy
time wondering why printer manufacturers are including chargen support
in firmware.
I have the same bewilderment about people allowing such unsolicited traffic into their network(s) in the first place. Even with IPv6 (where there's no NAT forcing the issue), I run a default deny policy... if nothing asked for it, it doesn't get in.
Also, why the hell aren't providers not doing anything to limit spoofing?!? I'll staring right at you AT&T (former Bellsouth.)
All of the above plus very poorly managed network / network
security. (sadly a Given(tm) for anything ending dot-e-d-u.) a) why
are *printers* given public IPs? and b) why are internet hosts
allowed to talk to them? I actually *very* surprised your printers
are still functional if the whole internet can reach them.
Who really has a solid motive to make them stop working (other than a
printer manufacturer who wants to sell them more) ?
Guess what, they have /16s, they use them, and they like
the ability to print from one side of campus to the other. Are you
suggesting gigantic NATs with 120,000 students and faculty behind them?
A per-building NAT would work, with static translations for printers
in that building, and an ACL with an allow list including IPsec
traffic to the printer from the campus' IP range.
They don't have to use NAT though to avoid unnecessary exposure of
services on internal equipment to the larger world.
I have a hard time blaming a school for this. I have an easy
time wondering why printer manufacturers are including chargen support
in firmware.
They probably built their printer on top of a general purpose or
embedded OS they purchased from someone else, or reused, that
included an IP stack -- as well as other features that were
unnecessary for their use case.
Or the chargen tool may have been used during stress tests to verify
proper networking, and that the IP stack processed bits without
corrupting them; with the manufacturer forgetting/neglecting to turn
off the unnecessary feature, forgetting to remove/disable that bit of
software, or seeing no need to, before mass producing.
Do you have any actual evidence that a .edu of (say) 2K employees
is statistically *measurably* less secure than a .com of 2K employees?
We keep hearing that meme - and yet, looking at the archives of this list,
I see a lot more stories of network providers who should know better doing
stupid stuff than I see of .edu's doing stupid stuff.
The Verizon report says small business is actually the biggest cesspit of abuse:
Duh, so people cannot print to them. (amungst various other creative pranks)
From a cybercriminal pov, to swipe the things you're printing... like that CC authorization form you just printed, or a confidential contract, etc. (also, in many offices, the printer is also the scanner and fax)
But seriously, how do you measure one's security? The scope is constantly changing. While there are companies one can pay to do this, those reports are *very* rarely published. And I've not heard of a single edu performing such an audit. The only statistics we have to run with are of *known* breaches. And that's a very bad metric as a company with no security at all that's had no (reported) intrusions appears to have very good security, while a company with extensive security looks very bad after a few breaches. One has noone sniffing around at all, while the other has teams going at it with pick-axes. One likely has noone in charge of security, while the other has an entire security department.
FWIW, last August we noticed 2.5Gbps of chargen being reflected off ~160
IPs (with large responses in violation of the RFC). As I recall, some
quick investigation indicated it was mostly printers. I notified several
of the worst offenders (rated by bandwidth).
While I think it's silly to be exposing chargen to the world (especially as
a default service in a printer!), the real problem here is networks that
allow spoofed traffic onto the public internet. In the rare cases we see
spoofed traffic I put special effort into tracing them to their source, and
then following up to educate those providers about egress filtering. I'd
appreciate it if others did the same.
