A couple of years ago I worked as a Linux system administrator at a small Internet service provider. Among all the regular system administrator duties I also had the privilege to write useful software and tools for Linux. One of my tasks was to write a utility to record how much traffic each of the customers was using.
The network diagram for this ISP was very simple. The gateway to the Internet was a powerful multi-core Linux box, which acted as a router and a firewall and also performed traffic shaping. Now it had to do traffic accounting as well.
Simplified network diagram. The gateway is a Linux box.
At that time, I had already mastered iptables and I had noticed that when listing the existing rules, iptables would display the packet count and the total byte count for each rule. I thought, why not use this for accounting? So I did. I created a script that creates an empty rule that always accepts traffic and passes it through the firewall for each user's IP address and another script that extracts the packet and byte count.
Here is a detailed explanation of how I did it exactly.
First, let's see what iptables shows us when we have just booted up.
# iptables -L -n -v -x
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Right after booting, there are no rules added and no traffic has passed through. Let's get familiar with the output that we'll be interested in when we have some rules. Notice the pkts and bytes columns? The 'pkts' stands for packets and displays the total number of packets matched by the rule. The 'bytes' stands for total number of bytes matched by the rule. Notice also three so called "chains" - INPUT, FORWARD and OUTPUT. The INPUT chain is for packets to the Linux box itself, OUTPUT chain is for packets leaving the Linux box (generated by programs running on the Linux box) and FORWARD is for packets passing through the box.
You might also be interested in the command line arguments that I used:
- -L lists all the rules.
- -n does not resolve the ip addresses.
- -v lists the packet and byte count.
- -x displays the byte count (otherwise it gets abbreviated to 200K, 3M, etc).
A firewall in production will have the FORWARD chain filled up with various entries already. To avoid messing them up, let's create a new traffic accounting chain called TRAFFIC_ACCT:
# <strong>iptables -N TRAFFIC_ACCT</strong>
# iptables -L -n -v -x
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
<strong>Chain TRAFFIC_ACCT (0 references)</strong>
pkts bytes target prot opt in out source destination
Now let's redirect all the traffic going through the machine to match the rules in the TRAFFIC_ACCT chain:
# <strong>iptables -I FORWARD -j TRAFFIC_ACCT</strong>
# iptables -L -n -v -x
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 TRAFFIC_ACCT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
<strong>Chain TRAFFIC_ACCT (0 references)</strong>
pkts bytes target prot opt in out source destination
If you have a Linux desktop computer, then you can insert the same rule in the INPUT chain (iptables -I INPUT -j TRAFFIC_ACCT) as the packets won't be routed and will be going to your computer.
The command line argument -L can actually take the name of a chain to list the rules from. From now on we will only be interested in rules of TRAFFIC_ACCT chain:
# iptables -L -n -v -x
Chain TRAFFIC_ACCT (1 references)
pkts bytes target prot opt in out source destination
Now to illustrate the main idea, we can play with the rules. For example, let's do the breakdown of traffic by tcp, udp and icmp protocols. To do that we insert three rules in the TRAFFIC_ACCT chain - one to match tcp protocol, one to match udp protocol, and the last one to match icmp protocol.
# iptables -A TRAFFIC_ACCT -p tcp # iptables -A TRAFFIC_ACCT -p udp # iptables -A TRAFFIC_ACCT -p icmp
After some time has passed, let's look at what we have:
# iptables -L TRAFFIC_ACCT -n -v -x
Chain TRAFFIC_ACCT (1 references)
pkts bytes target prot opt in out source destination
4356 2151124 tcp -- * * 0.0.0.0/0 0.0.0.0/0
119 15964 udp -- * * 0.0.0.0/0 0.0.0.0/0
3 168 icmp -- * * 0.0.0.0/0 0.0.0.0/0
We see that 4356 tcp packets totaling 2151124 bytes (2 megabytes) have passed through the firewall, 119 udp packets and 3 icmp packets.
You can zero out the counters with -Z iptables command:
# <strong>iptables -Z TRAFFIC_ACCT</strong>
# iptables -L TRAFFIC_ACCT -n -v -x
Chain TRAFFIC_ACCT (1 references)
pkts bytes target prot opt in out source destination
0 0 tcp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 udp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 icmp -- * * 0.0.0.0/0 0.0.0.0/0
You can remove all the rules from TRAFFIC_ACCT chain with -F iptables command:
# <strong>iptables -F TRAFFIC_ACCT</strong>
# iptables -L TRAFFIC_ACCT -n -v -x
Chain TRAFFIC_ACCT (1 references)
pkts bytes target prot opt in out source destination
Another fun example you can do is count how many actual connections have been made:
# <strong>iptables -A TRAFFIC_ACCT -p tcp --syn</strong>
# iptables -L -n -v -x
Chain TRAFFIC_ACCT (1 references)
pkts bytes target prot opt in out source destination
5 276 tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x16/0x02
Shows us that 5 tcp packets which start the connections have been sent. Pretty neat, isn't it?
What I did in my case is I added user IP addresses to the TRAFFIC_ACCT chain. Then, I periodically listed and recorded the traffic and zero'ed it out.
You can even create two chains TRAFFIC_ACCT_IN and TRAFFIC_ACCT_OUT to match incoming and outgoing traffic:
# iptables -N TRAFFIC_ACCT_IN
# iptables -N TRAFFIC_ACCT_OUT
# iptables -I FORWARD -i eth0 -j TRAFFIC_ACCT_IN
# iptables -I FORWARD -o eth0 -j TRAFFIC_ACCT_OUT
# iptables -L -n -v -x
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 TRAFFIC_ACCT_OUT all -- * eth0 0.0.0.0/0 0.0.0.0/0
0 0 TRAFFIC_ACCT_IN all -- eth0 * 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain TRAFFIC_ACCT_IN (1 references)
pkts bytes target prot opt in out source destination
Chain TRAFFIC_ACCT_OUT (1 references)
pkts bytes target prot opt in out source destination
For example, to record incoming and outgoing traffic usage of IP addresses 192.168.1.2 and 192.168.1.3 you can do:
# iptables -A TRAFFIC_ACCT_IN --dst 192.168.1.2 # iptables -A TRAFFIC_ACCT_IN --dst 192.168.1.3 # iptables -A TRAFFIC_ACCT_OUT --src 192.168.1.2 # iptables -A TRAFFIC_ACCT_OUT --src 192.168.1.2
And to list the rules:
# iptables -L TRAFFIC_ACCT_IN -n -v -x
Chain TRAFFIC_ACCT_IN (1 references)
pkts bytes target prot opt in out source destination
368 362120 all -- * * 0.0.0.0/0 192.168.1.2
61 9186 all -- * * 0.0.0.0/0 192.168.1.3
# iptables -L TRAFFIC_ACCT_OUT -n -v -x
Chain TRAFFIC_ACCT_OUT (1 references)
pkts bytes target prot opt in out source destination
373 22687 all -- * * 192.168.1.2 0.0.0.0/0
101 44711 all -- * * 192.168.1.3 0.0.0.0/0
That concludes it. You see that it is trivial to do accurate traffic accounting on a Linux machine. You can even visualize the traffic and create a web application that draws nice graphs.
You can output the data usage in a nice way with this combination of iptables and awk commands:
# iptables -L TRAFFIC_ACCT_IN -n -v -x | awk '$1 ~ /^[0-9]+$/ { printf "IP: %s, %d bytes\n", $8, $2 }'
IP: 192.168.1.2, 1437631 bytes
IP: 192.168.1.3, 449554 bytes
# iptables -L TRAFFIC_ACCT_OUT -n -v -x | awk '$1 ~ /^[0-9]+$/ { printf "IP: %s, %d bytes\n", $7, $2 }'
IP: 192.168.1.2, 88202 bytes
IP: 192.168.1.3, 244848 bytes
I first learned IPTables from this tutorial. It's the best tutorial you can find on the subject.
Next time I'll write about how I implemented traffic shaping with tc and iptables. See you then!