Rare TCP state 2 -- Single ESTABLISHED

Rare State 2

Circumstance

• Server listen on port with default backlog.

• Client initial a large number of connect at the same time.

Demo

sudo netstat -nap  | grep -w '5555' | awk '{print $6" "$7}' | sort | uniq -c
      1 LISTEN 20310/./Server
    125 ESTABLISHED -
     27 ESTABLISHED 20310/./Server
     69 SYN_RECV -
    221 ESTABLISHED 59253/./ClientSend
     79 SYN_SENT 59253/./ClientSend
    
sudo netstat -nap  | grep -w '5555' | awk '{print $6" "$7}' | sort | uniq -c
     54 ESTABLISHED -
    138 ESTABLISHED 20310/./Server
    300 ESTABLISHED 59253/./ClientSend
      1 LISTEN 20310/./Server

State

After a massive current connect (with a relatively small backlog)

[ESTABLISH (Client)| NULL (Server)]()

Fix

Solutions

• Disable SYN cookies
  For security reason, we don't want to do this.

• Increase the backlog size of listen socket.
  /proc/sys/net/ipv4/tcp_max_syn_backlog, default 2048, change to 8192

/proc/sys/net/core/somaxconn, default 128, change to 4096

sudo tail -f /var/log/messages

Backlog

man listen

• backlog ... it specifies the queue length for [completely established sockets]() waiting to be accepted, instead of the number of incomplete connection requests.

• If the backlog argument is greater than the value in /proc/sys/net/core/somaxconn, then it is silently truncated to that value; the default value in this file is 128. ...

• The maximum length of the queue for [incomplete sockets]() can be set using /proc/sys/net/ipv4/tcp_max_syn_backlog.

  When  syncookies  are enabled there is no logical maximum length and this setting is ignored.  See tcp(7) for more information.

This means that current Linux versions use two distinct queues:

• a SYN queue with a size specified by a system wide setting and

• SYN_RECV [cona1, cona2, cona3 ... ]

• an accept queue with a size specified by the application.

• ESTABLISHED [conb1, conb2, conb3 ... ]

SYN cookies

from wikipedia:
SYN cookie is a technique used to resist SYN flood attacks. ... ... defines SYN cookies as ["particular choices of initial TCP sequence numbers by TCP servers." ]()

In particular, the use of SYN cookies allows a server to avoid dropping connections when the SYN queue fills up. Instead, the server behaves as if the SYN queue had been enlarged.

The server sends back the appropriate SYN+ACK response to the client but discards the SYN queue entry.

If the server then receives a subsequent ACK response from the client, the server is able to reconstruct the SYN queue entry using information encoded in the TCP sequence number.

How the cookie is generated

net/ipv4/syncookies.c

How TCP backlog works in Linux
TCP SYN Cookies – DDoS defence
Quick Blind TCP Connection Spoofing with SYN Cookies

Question

• SynCookie can rebuild the connection , but in demo it did not.

• Can not find any warn log in dmesg or /var/log/messages

• The SYN_RECEIVE queue is not full

• SYN_RECV < SYN_SENT

sudo netstat -nap  | grep -w '5555' | awk '{print $6" "$7}' | sort | uniq -c
      1 LISTEN 20310/./Server
    125 ESTABLISHED -
     27 ESTABLISHED 20310/./Server
     69 SYN_RECV -
    221 ESTABLISHED 59253/./ClientSend
     79 SYN_SENT 59253/./ClientSend
    
sudo netstat -nap  | grep -w '5555' | awk '{print $6" "$7}' | sort | uniq -c
     54 ESTABLISHED -
    138 ESTABLISHED 20310/./Server
    300 ESTABLISHED 59253/./ClientSend
      1 LISTEN 20310/./Server

Did SynCookie take effect?

Demo
SynCookie is not triggered

Why Syn Cookie did not work?

The SYN_RECEIVE queue is not full

Ack Missing

Demo

Client	Server
closed	closed
closed	listen
(SYN)
syn_sent	syn_received
	(SYN, ACK)
established
[(ACK)missing]()
established	established

Why

1257 /*
1258  * The three way handshake has completed - we got a valid synack -
1259  * now create the new socket.
1260  */
1261 struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,
1262                                   struct request_sock *req,
1263                                   struct dst_entry *dst,
1264                                   struct request_sock *req_unhash,
1265                                   bool *own_req)
1266 {
1267         struct inet_request_sock *ireq;
1268         struct inet_sock *newinet;
1269         struct tcp_sock *newtp;
1270         struct sock *newsk;
1271 #ifdef CONFIG_TCP_MD5SIG
1272         struct tcp_md5sig_key *key;
1273 #endif
1274         struct ip_options_rcu *inet_opt;
1275
1276         if (sk_acceptq_is_full(sk))
1277                 goto exit_overflow;

• The code after the exit_overflow label will perform some cleanup, update the ListenOverflows and ListenDrops statistics in /proc/net/netstat and then return NULL.

• This will trigger the execution of the listen_overflow code in tcp_check_req:

774 listen_overflow:
775         if (!sysctl_tcp_abort_on_overflow) {
776                 inet_rsk(req)->acked = 1;
777                 return NULL;
778         }

This means that unless /proc/sys/net/ipv4/tcp_abort_on_overflow is set to 1 (in which case the code right after the code shown above will send a RST packet), the implementation basically does… nothing!

To summarize, if the TCP implementation in Linux receives the ACK packet of the 3-way handshake and the accept queue is full, it will basically [ignore that packet ]() .

SYN_RECV not full && SYN_RECV < SYN_SENT

sudo netstat -nap  | grep -w '5555' | awk '{print $6" "$7}' | sort | uniq -c
      1 LISTEN 20310/./Server
    125 ESTABLISHED -
     27 ESTABLISHED 20310/./Server
     69 SYN_RECV -
    221 ESTABLISHED 59253/./ClientSend
     79 SYN_SENT 59253/./ClientSend

The reason is the following code in the tcp_v4_conn_request function (which does the processing of SYN packets) in net/ipv4/tcp_ipv4.c:

/* Accept backlog is full. If we have already queued enough
         * of warm entries in syn queue, drop request. It is better than
         * clogging syn queue with openreqs with exponentially increasing
         * timeout.
         */
        if (sk_acceptq_is_full(sk) && inet_csk_reqsk_queue_young(sk) > 1) {
                NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_LISTENOVERFLOWS);
                goto drop;
        }

What this means is that if the accept queue is full, then the kernel will impose a limit on the rate at which SYN packets are accepted. If too many SYN packets are received, some of them will be dropped. In this case, it is up to the client to retry sending the SYN packet