$ cd ~/repos
$ git clone git://git.osmocom.org/libosmocore
$ cd libosmocore/
$ aclocal && autoreconf --install
$ ./configure --prefix=/opt/osmocom
$ gmake install

I am using relayd to relay incomming connections to thttpd server. Every http request is logged with IP address of the relayd server not with the real client IP address. Part of relayd.conf file:

...
http protocol httpssl {
        header append "$REMOTE_ADDR" to "X-Forwarded-For"
        header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By"
...

Relayd adds X-Forwarded-For and X-Forwarded-By fields into HTTP header. With the following patch thttpd will read them and real client IP will appear in log file.

...
[123.123.123.123] by [127.0.0.1:8443] - - [03/Dec/2013:15:50:48 +0200] "GET / HTTP/1.1" 200 857 "" \
"Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Firefox/17.0"
...

# tar zxvf thttpd-2.25b.tar.bz
# cp thttpd-log-proxy.patch thttpd-2.25b/
# cd thttpd-2.25b/
# patch < thttpd-log-proxy.patch
# ./configure
# make
# make install

I am using relayd for proxy but if you prefer you can use another software. Just make sure X-Forwarded-For and X-Forwarded-By will appear in the HTTP header.

$ vim 1.c

#include 

main()
{
        char *token[]={
                "prase",
                "kon",
                "guz"
        };

        int i;

        return 0;
}

$ gcc -g 1.c # with debug

$ gdb ./a.out

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
(gdb) break main
Breakpoint 1 at 0x4005b4: file 1.c, line 9.
(gdb) run
Starting program: /data/dev/tmp/a.out

Breakpoint 1, main () at 1.c:9
9               };
(gdb) info locals
token = {0x1 , 0x1 , 0x7fffffffd7e0 ""}
i = 0
(gdb) next
13              return 0;
(gdb) info locals
token = {0x400623 "prase", 0x400629 "kon", 0x40062d "guz"}
i = 0
(gdb) quit

$ gcc 1.c # without debug

$ gdb ./a.out

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)...
(gdb) break main
Breakpoint 1 at 0x4005b4
(gdb) run
Starting program: /data/dev/tmp/a.out
(no debugging symbols found)...(no debugging symbols found)...
Breakpoint 1, 0x00000000004005b4 in main ()
(gdb) info locals
No symbol table info available.
(gdb) disassem
Dump of assembler code for function main:
0x00000000004005b0 :    push   %rbp
0x00000000004005b1 :    mov    %rsp,%rbp
0x00000000004005b4 :    movq   $0x400623,-0x20(%rbp)
0x00000000004005bc :   movq   $0x400629,-0x18(%rbp)
0x00000000004005c4 :   movq   $0x40062d,-0x10(%rbp)
0x00000000004005cc :   mov    $0x0,%eax
0x00000000004005d1 :   leaveq
0x00000000004005d2 :   retq
0x00000000004005d3 :   nop
0x00000000004005d4 :   nop
0x00000000004005d5 :   nop
0x00000000004005d6 :   nop
0x00000000004005d7 :   nop
0x00000000004005d8 :   nop
0x00000000004005d9 :   nop
0x00000000004005da :   nop
0x00000000004005db :   nop
0x00000000004005dc :   nop
0x00000000004005dd :   nop
0x00000000004005de :   nop
0x00000000004005df :   nop
End of assembler dump.
(gdb) print (char *)0x400623
$12 = 0x400623 "prase"
(gdb) print (char *)0x400629
$13 = 0x400629 "kon"
(gdb) print (char *)0x40062d
$14 = 0x40062d "guz"

Just as reminder.

Get list of the IP addresses from /var/log/maillog which are trying to send mail to unknown user:

# grep -A1 "User unknown" /var/log/maillog | sed -n 's/.*\[\([0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\)].*/\1/p'

190.238.202.226
201.219.199.50
201.255.168.32
203.190.215.209
84.232.162.62
201.205.254.118
190.233.172.167
24.231.49.46
204.252.154.113
80.14.151.159
151.15.20.40
151.15.20.40
213.180.29.195
2.226.32.189
90.190.222.150
88.248.101.247
88.248.101.247

My spammers list getting bigger.

rssroll has been committed to FreeBSD ports tree. :)

Last attempts:

Apr 18 10:32:56 chaosophia spamd[65751]: 151.54.29.82: connected (1/0)
Apr 18 10:33:09 chaosophia spamd[65751]: (GREY) 151.54.29.82: <simmonsj71@buxrud.se> -> <gegjsd@koue.net>
Apr 18 10:33:09 chaosophia spamd[65751]: (GREY) 151.54.29.82: <simmonsj71@buxrud.se> -> <sawjks@koue.net>
Apr 18 10:33:09 chaosophia spamd[65751]: 151.54.29.82: disconnected after 13 seconds.
Apr 18 10:40:10 chaosophia spamd[65751]: 151.54.29.82: connected (1/0)
Apr 18 10:40:21 chaosophia spamd[65751]: (GREY) 151.54.29.82: <harebrainedf4@bmatter.com> -> <gegjsd@koue.net>
Apr 18 10:40:21 chaosophia spamd[65751]: (GREY) 151.54.29.82: <harebrainedf4@bmatter.com> -> <sawjks@koue.net>
Apr 18 10:40:22 chaosophia spamd[65751]: (GREY) 151.54.29.82: <harebrainedf4@bmatter.com> -> <neikgj@koue.net>
Apr 18 10:40:22 chaosophia spamd[65751]: (GREY) 151.54.29.82: <harebrainedf4@bmatter.com> -> <ywljzd@koue.net>
Apr 18 10:40:22 chaosophia spamd[65751]: 151.54.29.82: disconnected after 12 seconds.

After 3 days and 10 hours the result is 458 entries in the spamd table.

And the winners are:

...
Apr 16 04:18:54 j102 spamd[58928]: 178.216.50.60: disconnected after 3936 seconds.
...
Apr 16 08:17:27 j102 spamd[58928]: 78.186.75.116: disconnected after 580 seconds.
...
Apr 16 21:11:30 j102 spamd[58928]: 186.60.196.6: disconnected after 470 seconds.
...
Apr 18 01:38:59 j102 spamd[58928]: 190.20.2.201: disconnected after 550 seconds.
...
Apr 18 05:20:39 j102 spamd[58928]: 84.77.149.238: disconnected after 540 seconds.
...

Configuration: I am using sendmail for smtp server. In front of the smtp server there is configured spamd server in greylisted mode.

# cat /etc/rc.conf
...
obspamd_enable="YES"
obspamd_flags="-4 -G 23:4:864 -h mail.chaosophia.net -l 127.0.0.1 -n \"open yourself for chaos\" -S10 -s1 -v -w1"
...
# cat /etc/pf.conf
...
rdr pass on $ext_if proto tcp from  to 78.128.27.117 port 25 -> 127.0.0.1 port 25
rdr pass on $ext_if proto tcp from ! to 78.128.27.117 port 25 -> 127.0.0.1 port 8025
...

The problem: since 15 of April spamd.log and spamdb start to increase too fast.

# less /var/log/spamd.log
...
Apr 15 01:48:53 chaosophia spamd[65751]: 186.137.96.185: connected (1/0)
Apr 15 01:49:05 chaosophia spamd[65751]: (GREY) 186.137.96.185: <schedulesxc42@yhbia.com> -> <ymyhdu@koue.net>
Apr 15 01:49:05 chaosophia spamd[65751]: (GREY) 186.137.96.185: <schedulesxc42@yhbia.com> -> <vyvtlv@koue.net>
Apr 15 01:49:06 chaosophia spamd[65751]: (GREY) 186.137.96.185: <schedulesxc42@yhbia.com> -> <segiqb@koue.net>
Apr 15 01:49:06 chaosophia spamd[65751]: (GREY) 186.137.96.185: <schedulesxc42@yhbia.com> -> <zojois@koue.net>
Apr 15 01:49:06 chaosophia spamd[65751]: 186.137.96.185: disconnected after 13 seconds.
Apr 15 01:50:19 chaosophia spamd[65751]: 190.107.105.160: connected (1/0)
Apr 15 01:50:32 chaosophia spamd[65751]: (GREY) 190.107.105.160: <dinedocj22@mthai.com> -> <prrueh@koue.net>
Apr 15 01:50:32 chaosophia spamd[65751]: (GREY) 190.107.105.160: <dinedocj22@mthai.com> -> <wdqbzq@koue.net>
Apr 15 01:50:32 chaosophia spamd[65751]: (GREY) 190.107.105.160: <dinedocj22@mthai.com> -> <zsiuwq@koue.net>
Apr 15 01:50:32 chaosophia spamd[65751]: (GREY) 190.107.105.160: <dinedocj22@mthai.com> -> <xbsfua@koue.net>
Apr 15 01:50:33 chaosophia spamd[65751]: 190.107.105.160: disconnected after 14 seconds.
...
# spamdb | wc -l
794

794 is too big value for my idle smtp server so I have started to examine the spamd.log file. Every few minutes few different smtp servers were trying to connect to my smtp server and were trying to send mail to unknown user: prrueh, wdqbzq, zsiuwq. I have been decided to start another spamd server in blacklisted mode and will redirect all of the spammers to collect wasted minutes.

# cat /etc/pf.conf
...
rdr pass on $ext_if proto tcp from  to 78.128.27.117 port 25 -> 192.168.100.102 port 8025
...
# pfctl -t spamd -T add 190.107.105.160

Blacklisted spamd:

# cat /var/log/spamd.log
...
Apr 15 10:05:45 j102 spamd[58928]: 190.107.105.160 connected (1/0)
Apr 15 10:08:05 j102 spamd[58928]: 190.107.105.160 disconnected after 140 seconds.
...

Nice... but after few attempts another IP address have been appeared in the log file of the greylisted spamd server. So I created script which add them to the spamd table and run it from the cron.

#!/bin/sh

for i in `/usr/local/sbin/spamdb | tail | cut -d '|' -f 2  | sort | uniq`
do
	COUNT=`tail -n 100 /var/log/spamd.log | grep $i | grep GREY | wc -l`
	if [ $COUNT -gt 4 ]
	then
		pfctl -t spamd -T show | grep -q $i
		if [ $? -ne 0 ]
		then
			pfctl -t spamd -T add $i
		fi
	fi
done

Keep in mind that [ $COUND -gt 4 ] is a small value but it's OK to me. For 24 hours in my spammers list there is more than 200 entries.

It's not great solution but it works for now. Don't forget to check you log files regularly!

Case: Network A 192.168.102.0/24, network B 192.168.103.0/24 and Internet. Network A has access to Internet. Network B does NOT have access to Internet. Network B has access to network A. Need to start application in network B which uses service from Internet. Lets say the service is available on 78.128.27.117. Will use two servers CentOS release 6.3.

Configuration:

• IP: 192.168.102.63
• Mask: 255.255.255.0
• Gateway: 192.168.102.254

• IP: 192.168.103.133
• Mask: 255.255.255.0
• Gateway: 192.168.103.254

On 192.168.102.63:

# cat /etc/sysconfig/network-scripts/ifcfg-tun0

DEVICE=tun0
BOOTPROTO=none
ONBOOT=no
TYPE=GRE
PEER_OUTER_IPADDR=192.168.103.133
PEER_INNER_IPADDR=172.23.0.1
MY_INNER_IPADDR=172.23.0.2

# ifup tun0
# sysctl net.ipv4.conf.all.forwarding=1
# iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE

On 192.168.103.133:

# cat /etc/sysconfig/network-scripts/ifcfg-tun0

DEVICE=tun0
BOOTPROTO=none
ONBOOT=no
TYPE=GRE
PEER_OUTER_IPADDR=192.168.102.63
PEER_INNER_IPADDR=172.23.0.2
MY_INNER_IPADDR=172.23.0.1

# ifup tun0
# route add 78.128.27.117 gw 172.23.0.2

Nice ;)

Issue: Public DNS server, accessible from all over the world. Spoofed DNS requests flood victims networks.

# tcpdump -n -i sis0 port 53

14:59:33.030638 IP 178.33.159.24.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
14:59:33.033488 IP 78.128.27.117.53 > 178.33.159.24.25345: 10809 Refused- 0/0/1 (36)
14:59:36.069100 IP 37.59.194.196.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
14:59:36.071047 IP 78.128.27.117.53 > 37.59.194.196.25345: 10809 Refused- 0/0/1 (36)
14:59:37.718659 IP 178.33.159.24.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
14:59:37.720365 IP 78.128.27.117.53 > 178.33.159.24.25345: 10809 Refused- 0/0/1 (36)
14:59:39.413152 IP 178.33.159.24.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
14:59:39.414769 IP 78.128.27.117.53 > 178.33.159.24.25345: 10809 Refused- 0/0/1 (36)
14:59:39.861961 IP 37.59.194.196.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
14:59:39.863728 IP 78.128.27.117.53 > 37.59.194.196.25345: 10809 Refused- 0/0/1 (36)
14:59:40.360437 IP 94.23.145.254.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
14:59:40.362897 IP 78.128.27.117.53 > 94.23.145.254.25345: 10809 Refused- 0/0/1 (36)

178.33.159.24, 37.59.194.196, 94.23.145.254 are victims and the DNS server floods them with unrequested answers.

Workaround:

All requests are comming from port 25345. The DNS server is in separate jail and I use pf rdr to route the dns requests.

# cat /etc/pf.conf

...
set block-policy drop
...
no rdr on $ext_if proto udp from any port 25345 to 78.128.27.117 port 53
rdr pass on $ext_if proto udp from any to 78.128.27.117 port 53 -> 192.168.100.105
...
block all
...

# pfctl -n -f /etc/pf.conf
# pfctl -f /etc/pf.conf
# pfctl -s states | grep ":53)"

all udp 192.168.100.105:53 (78.128.27.117:53) <- 37.59.194.196:25345       MULTIPLE:MULTIPLE
all udp 192.168.100.105:53 (78.128.27.117:53) <- 178.33.159.24:25345       MULTIPLE:MULTIPLE
all udp 192.168.100.105:53 (78.128.27.117:53) <- 94.23.145.254:25345       MULTIPLE:MULTIPLE

# pfctl -k 192.168.100.105

States must be killed or replies will not stop.

1. Firewall policy is drop. Don't want to reply anything to the sender.
2. Don't redirect udp requests with source port 25345 and push them to the filter rules.
3. Block all.

Be careful with the other filter rules.

After workaround:

# tcpdump -n -i sis0 port 53

15:05:39.129676 IP 37.59.194.196.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
15:05:41.109744 IP 94.23.145.254.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
15:05:41.608512 IP 94.23.145.254.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
15:05:42.179105 IP 178.33.159.24.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
15:05:44.155841 IP 37.59.194.196.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
15:05:47.522229 IP 37.59.194.196.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
15:05:47.652616 IP 94.23.145.254.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
15:05:49.591880 IP 94.23.145.254.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
15:05:50.841837 IP 37.59.194.196.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
15:05:54.181441 IP 94.23.145.254.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)
15:05:54.323259 IP 178.33.159.24.25345 > 78.128.27.117.53: 10809+ [1au] ANY? isc.org. (36)