Hobbit DDOS Attack Detection (submission)
I had a hobbit-monitored site become the victim of a DDOS syn-flood attack. To help detect this in the futrue and also aid information-gathering, I did the following, which I am sharing:
- Made the following additions to hobbit-clients.cfg
Detect more than 100 half-open connections (possible syn-flood attack)
PORT STATE=SYN_RECV MIN=0 MAX=100 COLOR=red TRACK="SYN"
Detect more than 2000 established connections (possible simple DDOS
http get attack) PORT STATE=ESTABLISHED MIN=0 MAX=2000 COLOR=red
- Made a small modification to hobbit-linux.sh, specifically to the
[ports] section. I crammed it all into a single line.
echo "[ports]"
echo "SYN_REC Quick Stats:";SYNs=
netstat -pant 2>/dev/null | grep SYN | awk -F: '{print $2;}' |awk {'print $2"\t"$1'}| sort | uniq -c |sort -n;if [ -n "$SYNs" ]; then echo " # Address Port";printf "$SYNs\n";echo =============================;else echo "No SYNs Found";echo =============================;fi
Bug in RedHat's netstat spews annoying error messages.
netstat -ant 2>/dev/null
#1 allows Hobbit to detect and alert for 2 common DDOS attack signatures (syn floods and plain old http overloading), as well as creating graphs via the "track" feature.
#2 prefixes the netstat output you see in the "ports" column with a table of IPs that have half-open connections. It shows the address, IP, and what port they are "attacking". Here is an example (IPs and ports masked of course):
SYN_REC Quick Stats: # Address Port 1 289.122.3.20 80 1 213.102.135.60 80 1 200.120.152.6 8080 1 201.192.9.130 443 1 174.231.84.4 443 2 191.136.92.135 80 2 216.122.32.240 8080
The prefixing of this data in the client message has no effect on the Hobbit server, as it is only looking for the netstat output in the ports section, and so ignores that extra data (but still displays it). So if the site gets a syn flood, you can check the ports column of your web host(s) and easily see the offending IP(s) that are performing the attack. Note that most "floodbots" are coded such that they spoof their IP addresses, so the information you gather may not actually help you determine who is attacking, but is still useful for temporarily firewalling or whatever other steps you need to take to block it.
Note: this works on RHEL, so should work on RedHat/CentOS/Fedora. I havn't checked to see if debian/ubuntu has the same netstat parameters I used, so YMMV :)
-Charles
What kind of firewall do you have for your Hobbit? The firewall rules on my router detect these kinds of script kiddie attacks...
On 1/25/08, Charles Jones <jonescr at cisco.com> wrote:
I had a hobbit-monitored site become the victim of a DDOS syn-flood attack. To help detect this in the futrue and also aid information-gathering, I did the following, which I am sharing:
- Made the following additions to hobbit-clients.cfg
Detect more than 100 half-open connections (possible syn-flood attack)
PORT STATE=SYN_RECV MIN=0 MAX=100 COLOR=red TRACK="SYN"
Detect more than 2000 established connections (possible simple DDOS
http get attack) PORT STATE=ESTABLISHED MIN=0 MAX=2000 COLOR=red
- Made a small modification to hobbit-linux.sh, specifically to the [ports] section. I crammed it all into a single line. echo "[ports]" echo "SYN_REC Quick Stats:";SYNs=
netstat -pant 2>/dev/null | grep SYN | awk -F: '{print $2;}' |awk {'print $2"\t"$1'}| sort | uniq -c |sort -n;if [ -n "$SYNs" ]; then echo " # Address Port";printf "$SYNs\n";echo =============================;else echo "No SYNs Found";echo =============================;fiBug in RedHat's netstat spews annoying error messages.
netstat -ant 2>/dev/null
#1 allows Hobbit to detect and alert for 2 common DDOS attack signatures (syn floods and plain old http overloading), as well as creating graphs via the "track" feature.
#2 prefixes the netstat output you see in the "ports" column with a table of IPs that have half-open connections. It shows the address, IP, and what port they are "attacking". Here is an example (IPs and ports masked of course):
SYN_REC Quick Stats: # Address Port 1 289.122.3.20 80 1 213.102.135.60 80 1 200.120.152.6 8080 1 201.192.9.130 443 1 174.231.84.4 443 2 191.136.92.135 80 2 216.122.32.240 8080
The prefixing of this data in the client message has no effect on the Hobbit server, as it is only looking for the netstat output in the ports section, and so ignores that extra data (but still displays it). So if the site gets a syn flood, you can check the ports column of your web host(s) and easily see the offending IP(s) that are performing the attack. Note that most "floodbots" are coded such that they spoof their IP addresses, so the information you gather may not actually help you determine who is attacking, but is still useful for temporarily firewalling or whatever other steps you need to take to block it.
Note: this works on RHEL, so should work on RedHat/CentOS/Fedora. I havn't checked to see if debian/ubuntu has the same netstat parameters I used, so YMMV :)
-Charles
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
Josh Luthman wrote:
What kind of firewall do you have for your Hobbit? The firewall rules on my router detect these kinds of script kiddie attacks... I admin and consult on numerous Hobbit installations. This particular one is a small web cluster whose dedicated hosting provider has no IDS or decent firewall, as they get hit by these sorts of attacks and the only way to stop them is to request the blockage of specific IPs. I didn't mean to imply the "patch" I submitted was for everyone, just those who might find it useful :)
-Charles
Two different worlds, two different solutions =(
Thanks a whole bunch for the notes, though. I'm sure I'll come back to use it someday!
On 1/25/08, Charles Jones <jonescr at cisco.com> wrote:
Josh Luthman wrote:
What kind of firewall do you have for your Hobbit? The firewall rules on my router detect these kinds of script kiddie attacks... I admin and consult on numerous Hobbit installations. This particular one is a small web cluster whose dedicated hosting provider has no IDS or decent firewall, as they get hit by these sorts of attacks and the only way to stop them is to request the blockage of specific IPs. I didn't mean to imply the "patch" I submitted was for everyone, just those who might find it useful :)
-Charles
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
On Fri, Jan 25, 2008 at 01:59:08PM -0700, Charles Jones wrote:
I had a hobbit-monitored site become the victim of a DDOS syn-flood attack. To help detect this in the futrue and also aid information-gathering, I did the following [...]
Quite off-topic, but since this was on a Linux system you might want to look into enabling syn-cookies on these servers.
sysctl net.ipv4.tcp_syncookies=1
somewhere in a startup script will do that. It has no impact on the system unless the SYN connection queue fills up.
There's a thread discussing this on the Red Hat mailing list: http://www.redhat.com/archives/rhl-devel-list/2005-January/msg00447.html but the discussion is generic to Linux in all versions. Alan Cox (one of the main Linux kernel developers) also kills some of the myths about this setting in http://www.redhat.com/archives/rhl-devel-list/2005-January/msg00483.html
It won't help on the simple connection flooding (2000 active connections to your webserver), but there are other ways of stopping that - e.g. with Linux' built-in firewall (iptables): http://www.debian-administration.org/articles/187 The example is for fending off SSH brute-force attacks, but the same mechanism can be used for protecting webservers - although you will probably want to raise the limits from the 3 connections/second that the example allows.
Regards, Henrik
participants (3)
-
henrik@hswn.dk
-
jonescr@cisco.com
-
josh@imaginenetworksllc.com