Howdy,
Can anyone suggest tools, techniques and helpful contacts for
backtracking spoofed packets? At the moment someone is forging TCP
syns from my address block. I'm getting the syn/ack and icmp
unreachable backscatter. Enough that my service provider briefly
classified it a DDOS. I'd love to find the culprit.
Thanks,
Bill Herrin
Hi Bill,
can you post some forged packets please? You can send them offlist if you prefer.
It seems to be similar to what Octopus experience few weeks ago on this list.
Thanks
Jean St-Laurent | CISSP #634103
Check source interface for a flow from netflow. Good luck doing this
across multiple admin domains.
This is the issue (and the leitmotiv) of forged packets: you cannot
trace their origin in real world situation
It's not complete, but if you're receiving the ICMP net/port unreachable backscatter it should include a portion of the original packet. This might provide some insight into the TTL left on TCP the packet when it reached its destination which could provide a rough radius that you would need to look at. Also, if the TTL is constant it would support the idea that one or few hosts are spoofing your address block, but if the TTL varies widely it might indicate that many bots are spoofing your address block.
You might check looking glass tools or something like https://radar.qrator.net to see if someone is not only spoofing your address range, but has gone farther and has hijacked it.
Good luck,
--B
Transit providers can check their netflow and to identify the true source. Know any good mailing lists where transit providers hang out?
If you can share the victim IP and a timestamp, I may be able to offer additional advice off-list.
Damian
can you post some forged packets please? You can send them offlist if
you prefer.
Hi Jean,
Here are a couple examples (PDT this morning):
08:22:43.413250 IP (tos 0x0, ttl 55, id 10108, offset 0, flags [none],
proto ICMP (1), length 56)
45.89.93.26 > 199.33.225.218: ICMP host 45.89.93.26 unreachable -
admin prohibited filter, length 36
IP (tos 0x0, ttl 69, id 10108, offset 0, flags [DF], proto TCP
(6), length 40)
199.33.225.218.9851 > 45.89.93.26.443: [|tcp]
0x0000: 4500 0038 277c 0000 3701 28da 2d59 5d1a
0x0010: c721 e1da 030d 4b61 0000 0000 4500 0028
0x0020: 277c 4000 4506 dae4 c721 e1da 2d59 5d1a
0x0030: 267b 01bb a057 e903
08:25:47.787326 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto
TCP (6), length 44)
104.87.78.95.80 > 199.33.225.143.8667: Flags [S.], cksum 0xc97a
(correct), seq 1216155085, ack 11765035, win 29200, options [mss
1156], length 0
0x0000: 4500 002c 0000 4000 3606 e564 6857 4e5f
0x0010: c721 e18f 0050 21db 487d 0dcd 00b3 852b
0x0020: 6012 7210 c97a 0000 0204 0484
I have observed no consistency in the remote IP addresses. I receive
no more than a few of each and they don't line up with particular
networks. Remote ports are heavily 80, 443, 22, 25, etc. but a
smattering of less common ports too. I'm not seeing any RSTs at all
nor any port-unreachables. Lots of syn/acks and a few time exceeded
and host unreachables. I don't know what to make of that.
Hi Bill,
thanks for sharing the data. Indeed, I can’t offer you a way to backtrack the spoofed packets.
Anyway, I’m not sure what could you do legally as there is a very high chance that these people are not in the USA and the CFAA won’t apply to them.
Here is what I would do if I was in your situation.
Since these packets are spoof and malformed, I would block all SYN/ACK based on the length.
Depending on your hardware, it’s very easy to inspect only the SYN/ACK by length if you have modern hardware. On linux/unix, it’s also very straightforward. I’m not sure for windows though.
Here is the details of the analysis:
Today, all the SYN and SYN/ACK includes a minimum of options like MSS, WS, SACK, NOP (Only a spacer, sometimes 2) and extended TS. There might be others, but let’s use the basic one.
In your case, there are none. There is only MSS and the SYN length is 44 bytes. These are spoof packets maybe generated by either TCP-AMP like reported earlier.
I would try to block all SYN/ACK coming toward your network with a length of 44 bytes or lower. But, this is weird because it should be 54 bytes. Maybe there is some offloading of some sort in your gear.
Now depending on your hardware, it could work or it could kill your router as it will punt the cpu. I guess you have some modern gear.
What I do when I am not sure about the length, I start to accept and log at 60 bytes, then 58, 56, 54… 44 until I catch the gremlins.
Once you found the sweet spot, you drop all SYN/ACK toward your /23 lower than X bytes. It won’t kill or block anything legitimate if you do it properly. 
What will happen is that you will not reply to these spoof SYN/ACK with a RST and still allowing RST for legit SYN and SYN/ACK. Akamai and your service providers will be happy and should not penalize you.
I’m pretty sure that it will help you as it did for me in the past.
Let me know if it’s not clear and/or which part is foggy and I’ll try to give more details and better explanation.
Regards,
Jean St-Laurent
I don’t recommend filtering the SYN-ACK packets. That’s what Octolus did, and the result was leaving half-open SYN_RECV connections on all the nodes used for reflection. That has two downsides:
I don’t want to leave you with the impression that it’s hopeless… these attacks aren’t impossible to stop — it just requires convincing the transit providers to care.
Damian
Bill: I agree with Damian that you should try to ensure responding with RST to SYN/ACK; in fact, attackers are sometimes (often?) looking for networks that do not send RST in response to unsolicited SYN/ACK, to spoof their addresses in syn-flooding and other attacks (eg., syn-ack) against victim servers.
Not sending RST could even result in you receiving ICMP unreachable - esp. indicating filtering as you received - since server admins may have installed a filter against your prefix (to deal with such abuse). So, I wonder, it is possible that your network/FW/provider already filter the RST responses so they don’t reach the (victim) servers?
BTW, I’m covering these issues in my DoS lecture as part of the Net-Sec course (see URL below). The foils are available (although not yet latest version, will upload `soon’ ), the text of the net-sec (2nd) part - not so much, it may take me quite a while to make this (2nd) part useable.
I believe that Oculus blocked the RST and not the SYN/ACK.
It sounds the same but, it’s not.
I see 2 options here:
Continue to be DDoS and abuse. The result is maybe they will move on, but I doubt.
Try to block the malformed SYN/ACK and it will probably solve your issue. You have nothing to lose to try as you can easily fallback.
You can let us know what are the results and if I was wrong, I will publicly state that I was wrong.
But, make sure you do it properly. Do not block the RST, you need to block the malformed SYN/ACK before they hit your open reflectors.
Jean
Hi Amir,
To be clear: the majority of the addresses at my end are not
associated with live hosts. There's nothing there to respond.
My surprise about the lack of RSTs is the lack of RSTs from the remote
servers back to the addresses which have been spoofed. If the attacker
was hitting random ports on those hosts, I'd expect to see some RSTs.
If you happen to have decent netflow, try looking for packets sourced
from 199.33.224.0/24. You'll find a legitimate route in your tables
ending at AS11875 but today, at least, there are no legitimate packets
sourced from that address block.
Regards,
Bill
Bill said:
To be clear: the majority of the addresses at my end are not
associated with live hosts. There’s nothing there to respond.
I forgot to ask/mention , but that’s actually a common scenario. Of course in that case your router is supposed to respond with ICMP host unreachable which would have similar impact to RST (but I bet it doesn’t - which spec says to do it, many routers are not configured to do this). So attackers may idd have identified your address block as a good set of IP addresses to spoof when they attack different servers. It may help , if you configure your router to send ICMP unreachable (or RST, but that may be harder/less efficient, I think). Initially, this will cause the victim servers to stop re-sending you the syn/ack, so you’ll feel some relief, and also you’ll reduce the attack on these servers, which is A Good Deed. Eventually, hopefully, this may cause attackers to remove your IP prefix from their list of IP addresses which work well as spoofed source, but this will probably take quite a while. Sorry…
My surprise about the lack of RSTs is the lack of RSTs from the remote
servers back to the addresses which have been spoofed. If the attacker
was hitting random ports on those hosts, I’d expect to see some RSTs.
yes, but I bet attacker is not hitting random ports, attacker is hitting real servers in TCP listen.
(sorry don’t have time to netflow… have tons of work to do)
Regardless if you leave half-open SYN_RECV, you will still get abuse reports and blacklist it. These providers who blacklist and report you, are reporting you for bruteforce… and they only check for SYN_RECV.
Only solution we found was using SYNPROXY / CONNTRACK to validate.
In my opinion you can, just that the participants are not willing to do so due, in part, to the lack of acceptance of BCP38.
FWIW, Bellovin et al proposed an ICMP traceback mechanism in 2001
( https://tools.ietf.org/html/draft-ietf-itrace-04 ), but it seems
not to have progressed. Abstract:
It is often useful to learn the path that packets take through the
Internet, especially when dealing with certain denial-of-service
attacks. We propose a new ICMP message, emitted randomly by routers
along the path and sent randomly to the destination (to provide
useful information to the attacked party) or to the origin (to
provide information to decipher reflector attacks).