Tuning the TCP stack

IP defines the rules for IP addressing and routing packets over network and provides an identity IP address to each host on the network. TCP deals with the interconnection between two hosts and enables them to exchange data over network. TCP is a connection-oriented protocol and controls the ordering of packets, retransmission, error detection, and other reliability tasks.

New Linux kernel provides a tool called sysctl that can be used to modify kernel parameters at runtime without recompiling the entire kernel. We can use sysctl to modify and TCP/IP parameters to match our needs.

Set the maximum open files limit:

$ ulimit -n   # check existing limits for logged in user

ubuntu@ubuntu:/etc/haproxy$ ulimit -n
1024

# ulimit -n 65535   # root change values above hard limits

1. To permanently set limits for a user, open /etc/security/limits.conf and add the following lines at end of the file. Make sure to replace values in brackets, <>:

   <username>  soft  nofile  <value>     # soft limits
   <username>  hard  nofile  <value>     # hard limits
   

2. Save limits.conf and exit. Then restart the user session.

1. Set the TCP read and write buffers to 8 MB:

   # echo 'net.core.rmem_max=8388608' >> /etc/sysctl.conf
   # echo 'net.core.wmem_max=8388608' >> /etc/sysctl.conf
   

2. Increase the maximum TCP orphans:

   # echo 'net.ipv4.tcp_max_orphans=4096' >> /etc/sysctl.conf
   

3. Disable slow start after being idle:

   # echo 'net.ipv4.tcp_slow_start_after_idle=0' >> /etc/sysctl.conf
   

4. Minimize TCP connection retries:

   # echo 'net.ipv4.tcp_synack_retries=3' >> /etc/sysctl.conf
   # echo 'net.ipv4.tcp_syn_retries =3' >> /etc/sysctl.conf
   

5. Set the TCP window scaling:

   # echo 'net.ipv4.tcp_window_scaling=1' >> /etc/sysctl.conf
   

6. Enable timestamps:

   # echo 'net.ipv4.tcp_timestamp=1' >> /etc/sysctl.conf
   

7. Enable selective acknowledgements:

   # echo 'net.ipv4.tcp_sack=0' >> /etc/sysctl.conf
   

8. Set the maximum number of times the IPV4 packet can be reordered in the TCP packet stream:

   # echo 'net.ipv4.tcp_reordering=3' >> /etc/sysctl.conf
   

9. Send data in the opening SYN packet:

   # echo 'net.ipv4.tcp_fastopen=1'  >> /etc/sysctl.conf
   

10. Set the number of opened connections to be remembered before receiving acknowledgement:

    # echo 'tcp_max_syn_backlog=1500' >> /etc/sysctl.conf
    

11. Set the number of TCP keep-alive probes to send before deciding the connection is broken:

    # echo 'tcp_keepalive_probes=5' >> /etc/sysctl.conf
    

12. Set the keep-alive time, which is a timeout value after the broken connection is killed:

    # echo 'tcp_keepalive_time=1800' >> /etc/sysctl.conf
    

13. Set intervals to send keep-alive packets:

    # echo 'tcp_keepalive_intvl=60' >> /etc/sysctl.conf
    

14. Set to reuse or recycle connections in the wait state:

    # echo 'net.ipv4.tcp_tw_reuse=1' >> /etc/sysctl.conf
    # echo 'net.ipv4.tcp_tw_recycle=1' >> /etc/sysctl.conf
    

15. Increase the maximum number of connections:

    # echo 'net.ipv4.ip_local_port_range=32768 65535' >> /etc/sysctl.conf
    

16. Set TCP FIN timeout:

    # echo 'tcp_fin_timeout=60' >> /etc/sysctl.conf

Along with network parameters, tons of other kernel parameters can be configured with the sysctlcommand. The -a flag to sysctl will list all the available parameters:

$ sysctl -a

All these configurations are stored in a filesystem at the /proc directory, grouped in their respective categories. You can directly read/write these files or use the sysctl command:

ubuntu@ubuntu:~$ sysctl fs.file-max
fs.file-max = 98869
ubuntu@ubuntu:~$ cat /proc/sys/fs/file-max
98869