Identifying and Investigating Network-Bound Software in a Linux Server

Identifying and Investigating Network-Bound Software in a Linux Server




In this article we will learn how to correlate open ports with software running in a Linux server and understand why this knowledge is basic to operating and maintaining a obtain ecosystem. When a server is compromised, the consequence is not always as rash as complete data loss. Often the hacker will use the compromised great number to perpetrate his dominant goal, which is maintaining anonymity. One method to unprotected to this is install and function software which proxies network traffic. Due to this, the ability to generate a list of network-bound software and audit each is important.

This information is not limited to analyzing compromised or servers with security issues, as understanding what software running in your server that accepts input over the network (internet) is vital to keeping your server obtain.

After ssh’ing to our server as root, lets check which software is listening on tcp sockets.

# netstat -alntp grep LIST
tcp 0 0 10.100.10.10:322 0.0.0.0:* LISTEN 9653/sshd
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 8118/mysqld
tcp 0 0 10.100.10.10:80 0.0.0.0:* LISTEN 5266/httpd
tcp 0 0 10.100.10.10:80 0.0.0.0:* LISTEN 15404/openvpn
tcp 0 0 0.0.0.0:10000 0.0.0.0:* LISTEN 10065/perl
tcp 0 0 10.100.10.10:18081 0.0.0.0:* LISTEN 5266/httpd
tcp 0 0 127.0.0.1:8118 0.0.0.0:* LISTEN 23952/privoxy
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 9301/exim4
tcp 0 0 127.0.0.1:9050 0.0.0.0:* LISTEN 9664/tor
#

Each row represents an open socket, possibly to the internet. The fourth column tell us the IP address and port on our server the software is listening for connections on. 0.0.0.0 method all IP addresses. If the server does not have a firewall, one can assume that packets from the internet will be processed by the software listening on the appropriate port. already with a firewall, it is worthwhile to ensure that software is obtain. This is because software firewalls can be disabled/malfunction and hardware firewalls can be removed from the network without notice. The last column is in the format X/Y. X is the time of action ID or PID of the network-bound software. Y represents the name of the running program (this “name” is controllable by the program itself, and consequently cannot be relied upon to know what that program truly is). It is basic that you are able to correlate an open socket in your server with a program (PID) in your Linux server.

The first row tells us that “sshd” is running at PID 9653 and listening on IP address 10.100.10.10 port 322. This method tcp connections to 10.100.10.10 port 322 are handled by “sshd”. We can see that “httpd” is listening on ports 80 and 18081. The first port makes sense, this is the default port for http traffic. Port 18081 is a little strange, as it is an uncommon port number. To verify the authenticity of PID 5266, we can start by running this command to determine the absolute path to the binary which is network bound:

# ls -al /proc/5266/exe
lrwxrwxrwx 1 root root 0 Jun 24 20:34 /proc/5266/exe -> /usr/sbin/httpd

This tells us “httpd” is truly /usr/sbin/httpd. This makes sense because this is the normal path and name of the Apache webserver in CentOS. To verify that /usr/sbin/httpd is the true Apache binary, we can run two RPM commands:

# rpm -qf /usr/sbin/httpd
httpd-2.2.3-31.el5.centos.2
# rpm -V httpd-2.2.3-31.el5.centos.2
S.5….T c /etc/httpd/conf/httpd.conf

The first command uses flags “-qf” to query a file; the output is the package which the file belongs to. We then use the “-V” flag to verify the authenticity of that package in the system. The output consists of any alternation files. In this case, we learn that httpd.conf has been alternation. This makes sense and is shared, as the default apache configuration is not very useful for most organizations.

If /usr/sbin/httpd had been replaced with another binary, we see an immediate sign that the server could be compromised. We will simulate this by placing another binary in place of httpd.

# mv /usr/sbin/httpd /usr/sbin/httpd.real
# cp /bin/ls /usr/sbin/httpd
# rpm -V httpd-2.2.3-31.el5.centos.2
S.5….T c /etc/httpd/conf/httpd.conf
S.5….T /usr/sbin/httpd

In this case, /usr/sbin/httpd fails in three areas (periods represent passed tests):

S is the file size.
5 is the MD5 checksum of the file.
T is the alteration time of the file.
We can see that it makes sense for /etc/httpd/conf/httpd.conf to fail S and T, since the file was alternation (alteration time and file size changed). A failed md5 checksum in this case tells us that /usr/sbin/httpd is not the file which Centos provided us with.

Going back to our netstat report, we find this open socket:

tcp 0 0 10.100.10.10:18081 0.0.0.0:* LISTEN 5266/httpd

We will now check out this “httpd”:

# ls -al /proc/5266/exe
lrwxrwxrwx 1 root root 0 Jun 25 05:29 /proc/20424/exe -> /tmp/.var/lib/httpd

This is a rogue course of action, as “programs” should not be stored in /tmp in any normal Linux server. This file should not belong to any package, as rpm can verify for us:

#rpm -qf /tmp/.var/lib/httpd
file /tmp/.var/lib/httpd is not owned by any package

At this point, we know there is some insecurity in this server, as sockets are open by software which we did not install. Tracing the source of this rogue network program is another topic. Deleting the file may solve the immediate issue but rest assured the file and course of action will return without solving the inner insecurity.

We should continue to audit each network bound course of action in our server. PIDs 23952 and 9664 are of particular interest based on the time of action names. Network connections can also be established via UDP sockets. To generate a list of udp ports bound to programs running in your server, we can use command:

# netstat -alnp grep udp
udp 0 0 0.0.0.0:53 0.0.0.0:* 18443/dnscache
udp 0 0 10.100.1.100:53 0.0.0.0:* 19029/named

The same procedure in identifying the responsible binary via the PID can be used.

When provisioning a new server, it is basic to understand and research each open socket. Disabling unnecessary network bound software is basic in limiting the entry points to the server, consequently increasing security. After we clarify each piece of software which may be exploited, we should check the version and configuration of that software to ensure it is not unprotected to any known security issues.




leave your comment

Top