Mapping sockets of a process to the remote end point

Recently, one of our long running processes started exhibiting a high number of open file handles. We were leaking handles somewhere. The first thing is to figure out what handles are open, which is easy in Linux with /proc. Just plug in the PID of your process:

ls -ltr /proc/21657/fd

This spits out all the open file handles for the process with PID 21657. Here is an example of an open socket:

lrwx------ 1 user user 64 May 20 13:20 649 -> socket:[2336308491]

This alone doesn't tell us much. Our application use sockets for many reasons. There are connections to mysql, memcache and mongodb. There are sockets listening and responding to requests. There are connections made to web servers.

To get an idea of the two end points of the socket, we need to look at /proc/net/tcp (as well as tcp6, udp, udp6) :

user@host ~$ cat /proc/net/tcp6 | grep 2336308491
 129: 0000000000000000FFFF00004D29650A:9030 0000000000000000FFFF00005881754A:01BB 08 00000000:00000001 00:00000000 00000000   237        0 2336308491 1 ffff81061cf60740 371 40 0 4 -1

This is a connection to a SSL port on the remote end; 0x01BB = 443