Traffic Accounting using iptables

Introduction

Previosuly we have been using ipfm for traffic accounting but moving up to gigabit network and newer kernels ipfm have been giving us problems. Looking at alterlatives there are various tools and guides on and for traffic accounting using Linux, none of which I found really maching my needs.

The following text will describe the setup I build in order to get the traffic statistics I wanted.

The primary parts are:

iptables module

I've build a xtables match module ipt_traffic which based on a netaddress and netmask optionally counts traffic amounts per IP, connections per port and/or connections per IP. These values are then made available through proc.

Example

iptables -A PROMISC -m traffic --traffic --netmask 255.255.240.0 --netbase 193.11.176.0 --nointernal --name RSN

This rule will count traffic amounts per IP in and out from 193.11.176.0/20 (though not inbetween machines in different subnets as given by --nointernal). The traffic data is then available through /proc/net/xt_traffic/RSN which is on the format

iptables -A PROMISC -m traffic --traffic --portconn --ipconn --netmask 255.255.240.0 --netbase 193.11.176.0 --nointernal --name RSN

This rule will count traffic amounts per IP as well as count connections per port and IP in and out from 193.11.176.0/20 (though not inbetween machines in different subnets as given by --nointernal). The traffic data is then available through files in /proc/net/xt_traffic/.

Parameters

parameterDescription
--trafficCount traffic per IP (divided over TCP/UDP/OTHER)
--portconnCount connections per port (divided over TCP/UDP/ICMP/OTHER)
--ipconnCount connectiosn per IP (divided over TCP/UDP/ICMP/OTHER)
--netmask xxx.xxx.xxx.xxxThe netmask for the network we want to monitor
--netbase xxx.xxx.xxx.xxxThe base for the network we want to monitor
--name [name]Name for the file in the procfs
--hashsize [size]To override the hash size used to keep the traffic data (default is based on the netsize given)
--clearonreadClear records as they are read from proc.
[!] --nointernalDo not count traffic between nodes within the ipnet (or only count traffic between internal nodes)

Format of files in proc

Traffic amounts will be represented as a tabseparated list for each IP as seen below.

        IP              TOTAL           TCP             UDP             OTHER
        193.11.185.xxx  17489073        156686  17488837        156686  0       0       236     0
        193.11.185.xxx  414931  94048   406613  83229   8318    10819   0       0                
        193.11.185.xxx  7883982 15214658        41118   101703  7842864 15112955        0       0
        193.11.177.xxx  0       171721  0       4418    0       76      0       167227           
        193.11.185.xxx  1152329 38232531        1125869 38223727        22240   8804    4220    0
        193.11.185.xxx  4777835 96993266        2162599 95516511        2609847 1457634 5389    19121
        193.11.185.xxx  10120335        191629118       9995295 191525554       117907  103564  7133    0
        193.11.185.xxx  52157236        2257806 51809291        868785  346429  1388493 1516    528      
        193.11.185.xxx  1511600 147232  1511530 146862  70      370     0       0                        
        193.11.185.xxx  663084  359674  373285  75869   281267  275345  8532    8460                     
        193.11.185.xxx  10565993        368656  10565993        368656  0       0       0       0        
        193.11.185.xxx  335389  8443531 283312  8439889 52077   3642    0       0                        
        193.11.177.xxx  69581   348974  68319   347567  891     1239    371     168                      
        193.11.185.xxx  100705  3117951 100020  3117506 685     445     0       0                        
        ...
        

Per IP connections will be represented as a tabseparated list for each IP as seen below.

        IP              TOTAL           TCP             UDP             ICMP            OTHER
        172.20.0.2      615     3       1       1       0       2       614     0       0       0
        172.20.0.1      0       615     0       1       0       0       0       614     0       0
        

Per port connections will be represented as a tabseparated list for each port as seen below. ICMP and other will be shown as port 0.

        PORT            TOTAL           TCP             UDP             ICMP            OTHER
        0       313     313     0       0       0       0       313     313     0       0
        22      1       2       1       2       0       0       0       0       0       0
        53      0       2       0       0       0       2       0       0       0       0
        

Building and installing the module and extension

            # git clone http://git.pappkartong.bsnet.se/ipt_traffic.git
            # cd ipt_traffic
            # ./build build
            # mv libxt_traffic.so /usr/local/libexec/xtables/
            # mv ipt_traffic.ko /lib/modules/`uname -r`/kernel/net/ipv4/netfilter/
            # depmod -A
            # modprobe ipt_traffic
            # iptables -m traffic -h
        

Linux Kernel/iptables Modifications

As I wanted to run the traffic accounting on a separate box ona mirror/monitor port the kernel needed some modifications to forward promiscuous mode traffic to iptables.

The solution I found was in some patches from Deep Network Analyzer (DNA) adding the PROMISC hook to iptables.

Unfortunatly those patches are as of writing about three years old, but manually I was able to apply them to the 2.6.30 kernel and iptables 1.4.5

The kernel

Affected files

Building

            # cd linux-2.6.30.9
            # cat /path/to/patches/linux/*.diff|patch -u -p0
            # cp /boot/config-2.6.30-2-486 .config
            # make oldconfig
            # make
            # make modules_install
            # cp arch/x86/boot/bzImage /boot/vmlinuz-2.6.30-promisc
            # cp System.map /boot/System.map-2.6.30-promisc
            # mkinitramfs -o /boot/initrd.img-2.6.30-promisc 2.6.30.9
            # update grub
            # reboot
            

iptables

Affected files

Building

            # cd iptables-1.4.5/
            # cat /path/to/patches/iptables/*.diff|patch -u -p0
            # ./configure
            # make
            # make install
            

TODO

Download

git: # git clone http://git.pappkartong.bsnet.se/ipt_traffic.git

Other Projects

Other projects can be found at projects.pappkartong.se.