How do I remove a CLOSE_WAIT socket connection
I have written a small program that interacts with a server on a specific port. The program works fine, but:
Once the program terminated unexpectedly, and ever since that socket connection is shown in
CLOSE_WAIT state. If I try to run a program it hangs and I have to force it close, which accumulates even more
CLOSE_WAIT socket connections.
Is there a way to flush these connections?
CLOSE_WAIT means your program is still running, and hasn't closed the socket (and the kernel is waiting for it to do so). Add
netstat to get the pid, and then kill it more forcefully (with
SIGKILL if needed). That should get rid of your
CLOSE_WAIT sockets. You can also use
ps to find the pid.
SO_REUSEADDR is for servers and
TIME_WAIT sockets, so doesn't apply here.
Read more... Read less...
As described by Crist Clark.
CLOSE_WAIT means that the local end of the connection has received a FIN from the other end, but the OS is waiting for the program at the local end to actually close its connection.
The problem is your program running on the local machine is not closing the socket. It is not a TCP tuning issue. A connection can (and quite correctly) stay in CLOSE_WAIT forever while the program holds the connection open.
Once the local program closes the socket, the OS can send the FIN to the remote end which transitions you to LAST_ACK while you wait for the ACK of the FIN. Once that is received, the connection is finished and drops from the connection table (if your end is in CLOSE_WAIT you do not end up in the TIME_WAIT state).
I'm also having the same issue with a very latest Tomcat server (7.0.40). It goes non-responsive once for a couple of days.
To see open connections, you may use:
sudo netstat -tonp | grep jsvc | grep --regexp="127.0.0.1:443" --regexp="127.0.0.1:80" | grep CLOSE_WAIT
As mentioned in this post, you may use
/proc/sys/net/ipv4/tcp_keepalive_time to view the values. The value seems to be in seconds and defaults to 7200 (i.e. 2 hours).
To change them, you need to edit
Open/create `/etc/sysctl.conf` Add `net.ipv4.tcp_keepalive_time = 120` and save the file Invoke `sysctl -p /etc/sysctl.conf` Verify using `cat /proc/sys/net/ipv4/tcp_keepalive_time`
Even though too much of CLOSE_WAIT connections means there is something wrong with your code in the first and this is accepted not good practice.
You might want to check out: https://github.com/rghose/kill-close-wait-connections
What this script does is send out the ACK which the connection was waiting for.
This is what worked for me.
It should be mentioned that the
Socket instance in both client and the server end needs to explicitly invoke
close(). If only one of the ends invokes
close() then too, the socket will remain in CLOSE_WAIT state.
You can forcibly close sockets with
ss command; the
ss command is a tool used to dump socket statistics and displays information in similar fashion (although simpler and faster) to netstat.
To kill any socket in CLOSE_WAIT state, run this (as root)
$ ss --tcp state CLOSE-WAIT --kill