Previosuly we have been using ipfm for traffic accounting but moving up to gigabit network and newer kernels ipfm have been giving us problems. Looking at alterlatives there are various tools and guides on and for traffic accounting using Linux, none of which I found really maching my needs.
The following text will describe the setup I build in order to get the traffic statistics I wanted.
The primary parts are:
I've build a xtables match module ipt_traffic which based on a netaddress and netmask optionally counts traffic amounts per IP, connections per port and/or connections per IP. These values are then made available through proc.
iptables -A PROMISC -m traffic --traffic --netmask 255.255.240.0 --netbase 193.11.176.0 --nointernal --name RSN
This rule will count traffic amounts per IP in and out from 193.11.176.0/20 (though not inbetween machines in different subnets as given by --nointernal). The traffic data is then available through /proc/net/xt_traffic/RSN which is on the format
iptables -A PROMISC -m traffic --traffic --portconn --ipconn --netmask 255.255.240.0 --netbase 193.11.176.0 --nointernal --name RSN
This rule will count traffic amounts per IP as well as count connections per port and IP in and out from 193.11.176.0/20 (though not inbetween machines in different subnets as given by --nointernal). The traffic data is then available through files in /proc/net/xt_traffic/.
| parameter | Description |
|---|---|
| --traffic | Count traffic per IP (divided over TCP/UDP/OTHER) |
| --portconn | Count connections per port (divided over TCP/UDP/ICMP/OTHER) |
| --ipconn | Count connectiosn per IP (divided over TCP/UDP/ICMP/OTHER) |
| --netmask xxx.xxx.xxx.xxx | The netmask for the network we want to monitor |
| --netbase xxx.xxx.xxx.xxx | The base for the network we want to monitor |
| --name [name] | Name for the file in the procfs |
| --hashsize [size] | To override the hash size used to keep the traffic data (default is based on the netsize given) |
| --clearonread | Clear records as they are read from proc. |
| [!] --nointernal | Do not count traffic between nodes within the ipnet (or only count traffic between internal nodes) |
Traffic amounts will be represented as a tabseparated list for each IP as seen below.
IP TOTAL TCP UDP OTHER
193.11.185.xxx 17489073 156686 17488837 156686 0 0 236 0
193.11.185.xxx 414931 94048 406613 83229 8318 10819 0 0
193.11.185.xxx 7883982 15214658 41118 101703 7842864 15112955 0 0
193.11.177.xxx 0 171721 0 4418 0 76 0 167227
193.11.185.xxx 1152329 38232531 1125869 38223727 22240 8804 4220 0
193.11.185.xxx 4777835 96993266 2162599 95516511 2609847 1457634 5389 19121
193.11.185.xxx 10120335 191629118 9995295 191525554 117907 103564 7133 0
193.11.185.xxx 52157236 2257806 51809291 868785 346429 1388493 1516 528
193.11.185.xxx 1511600 147232 1511530 146862 70 370 0 0
193.11.185.xxx 663084 359674 373285 75869 281267 275345 8532 8460
193.11.185.xxx 10565993 368656 10565993 368656 0 0 0 0
193.11.185.xxx 335389 8443531 283312 8439889 52077 3642 0 0
193.11.177.xxx 69581 348974 68319 347567 891 1239 371 168
193.11.185.xxx 100705 3117951 100020 3117506 685 445 0 0
...
Per IP connections will be represented as a tabseparated list for each IP as seen below.
IP TOTAL TCP UDP ICMP OTHER
172.20.0.2 615 3 1 1 0 2 614 0 0 0
172.20.0.1 0 615 0 1 0 0 0 614 0 0
Per port connections will be represented as a tabseparated list for each port as seen below. ICMP and other will be shown as port 0.
PORT TOTAL TCP UDP ICMP OTHER
0 313 313 0 0 0 0 313 313 0 0
22 1 2 1 2 0 0 0 0 0 0
53 0 2 0 0 0 2 0 0 0 0
# git clone http://git.pappkartong.bsnet.se/ipt_traffic.git
# cd ipt_traffic
# ./build build
# mv libxt_traffic.so /usr/local/libexec/xtables/
# mv ipt_traffic.ko /lib/modules/`uname -r`/kernel/net/ipv4/netfilter/
# depmod -A
# modprobe ipt_traffic
# iptables -m traffic -h
As I wanted to run the traffic accounting on a separate box ona mirror/monitor port the kernel needed some modifications to forward promiscuous mode traffic to iptables.
The solution I found was in some patches from Deep Network Analyzer (DNA) adding the PROMISC hook to iptables.
Unfortunatly those patches are as of writing about three years old, but manually I was able to apply them to the 2.6.30 kernel and iptables 1.4.5
# cd linux-2.6.30.9
# cat /path/to/patches/linux/*.diff|patch -u -p0
# cp /boot/config-2.6.30-2-486 .config
# make oldconfig
# make
# make modules_install
# cp arch/x86/boot/bzImage /boot/vmlinuz-2.6.30-promisc
# cp System.map /boot/System.map-2.6.30-promisc
# mkinitramfs -o /boot/initrd.img-2.6.30-promisc 2.6.30.9
# update grub
# reboot
# cd iptables-1.4.5/
# cat /path/to/patches/iptables/*.diff|patch -u -p0
# ./configure
# make
# make install
git: # git clone http://git.pappkartong.bsnet.se/ipt_traffic.git
Other projects can be found at projects.pappkartong.se.