HTB: APTNightmare2

Challenge Overview

APTNightmare2 is a Hack The Box Sherlock challenge that plunges you into the aftermath of a sophisticated APT intrusion. The scenario states that upon completing a server recovery process, the incident response team uncovered persistent traffic, covert communication channels, and resilient processes that resisted termination efforts. The scope of the incident clearly exceeds the initial breach of servers and clients, and your task as a forensic investigator is to illuminate the shadows concealing these clandestine activities.

We are provided with a ZIP archive containing a Linux memory dump image along with the necessary Volatility profile for analysis. Given the nature of the scenario — persistent processes that refuse to die, hidden communication channels, and the fact that our only artifact is a raw memory dump — we can immediately infer that we are dealing with a rootkit. Rootkits operate at the kernel level, intercepting system calls and manipulating kernel data structures to hide their presence from userland tools. Memory forensics is the most reliable technique to unmask such threats, as the rootkit cannot easily tamper with the memory image once it has been captured.

Volatility Profile Setup

Before we can begin analyzing the memory dump, we need to configure Volatility with the correct Linux profile. The profile maps kernel symbol addresses and data structure offsets specific to the kernel version that was running on the compromised system. Without the correct profile, Volatility cannot interpret the memory structures properly. Fortunately, the required profile file is included in the challenge ZIP archive.

Adding the Profile

The profile needs to be copied into Volatility's Linux overlay directory. This directory contains custom profiles that extend Volatility's built-in support for various Linux kernel versions. Once the profile file is in place, Volatility can parse the memory dump's kernel structures correctly, enabling us to use Linux-specific plugins like linux_psscan, linux_check_modules, and linux_netscan.

# Copy the profile to Volatility's Linux overlay directory
cp Ubuntu_5.3.0-70-generic.zip /opt/volatility/plugins/overlays/linux/

# Verify the profile was added correctly
volatility --info | grep Ubuntu_5_3_0_70

Initial Evidence Collection

Before diving into specific questions, it is good practice to run several basic Volatility plugins and save their output to individual files. This approach allows us to quickly reference results without having to re-run the often time-consuming memory analysis commands. Key plugins to run initially include linux_psscan, linux_pslist, linux_netscan, linux_check_modules, and linux_enumerate_files. Having these outputs readily available saves significant time as we progress through the investigation.

# Run baseline collection of Volatility outputs
volatility -f memory.img --profile=LinuxUbuntu_5_3_0_70-genericx64 linux_pslist > pslist.txt
volatility -f memory.img --profile=LinuxUbuntu_5_3_0_70-genericx64 linux_psscan > psscan.txt
volatility -f memory.img --profile=LinuxUbuntu_5_3_0_70-genericx64 linux_netscan > netscan.txt
volatility -f memory.img --profile=LinuxUbuntu_5_3_0_70-genericx64 linux_check_modules > check_modules.txt
volatility -f memory.img --profile=LinuxUbuntu_5_3_0_70-genericx64 linux_enumerate_files > enum_files.txt

Task 1 — Reverse Shell Connection

The first task asks us to identify the IP address and port of the attacker's reverse shell. Since we are looking for network connections, the logical starting point is the linux_netscan plugin output. This plugin scans the memory image for network socket structures and reconstructs active connections, listening sockets, and established sessions that existed at the time of the memory capture.

When reviewing the netscan output, we need to look for suspicious outbound connections — particularly those originating from shell processes like bash. A reverse shell typically manifests as an established TCP connection from a bash or similar process connecting out to an external IP address. In legitimate systems, bash processes rarely initiate network connections, so any such connection is inherently suspicious and warrants immediate investigation.

$ volatility -f memory.img --profile=LinuxUbuntu_5_3_0_70-genericx64 linux_netscan

Immediately, a suspicious connection stands out in the output. A bash process with PID 3633 has an established TCP connection to an external IP address on port 443. This is a textbook reverse shell indicator — the attacker likely chose port 443 because HTTPS traffic is commonly allowed through firewalls and blends in with legitimate encrypted web traffic. The use of bash as the connecting process confirms this is a command shell being piped over the network, providing the attacker with interactive access to the compromised system.

Task 2 — Hidden Parent Process

The next task requires us to identify the true parent process ID (PPID) of the reverse shell. To find this, we first examine the linux_pslist output, filtering for the PID we discovered in the previous task (PID 3633). The pslist plugin walks the kernel's linked list of active processes, which is the same list that tools like ps and top read from. However, rootkits can manipulate this list to hide processes.

When we look up PID 3633 in the pslist output, the PPID appears as 1, which belongs to the init process. While it is possible for processes to be re-parented to PID 1 when their true parent exits, this is highly suspicious in the context of a reverse shell. An attacker's shell would typically be spawned by an exploit process, not directly by init. Furthermore, the pslist shows multiple bash processes, but none of them have a PPID that logically matches as the parent of the reverse shell.

$ grep 3633 pslist.txt
0xffffffff8a012300 bash            3633    1       1      0x0000000000000000 0x0

The PPID of 1 is suspicious because it suggests the parent process is either being misrepresented or intentionally hidden from the active process list. This is a classic rootkit technique — the rootkit unlinks the parent process from the kernel's task list, making it invisible to pslist and any tool that reads from that list. However, the process still exists in memory; it has simply been removed from the linked list.

Uncovering Hidden Processes with psscan

The linux_psscan plugin takes a fundamentally different approach: instead of walking the kernel's linked list, it scans the entire memory image for process structures by looking for specific kernel data structure signatures. This cross-view detection technique can find processes that have been unlinked from the active list by a rootkit. When we run psscan, we discover a previously hidden bash process that was not visible in the pslist output.

$ volatility -f memory.img --profile=LinuxUbuntu_5_3_0_70-genericx64 linux_psscan | grep bash
0xffffffff8a011c00 bash            3623   3622       1      0x0000000000000000 0x0
0xffffffff8a012300 bash            3633   3623       1      0x0000000000000000 0x0

The psscan output reveals the hidden bash process with PID 3623, which is the true parent of our reverse shell (PID 3633). This process was deliberately unlinked from the kernel's task list by the rootkit, but its memory structures remain intact and discoverable through scanning. The discrepancy between pslist and psscan is a definitive indicator of rootkit activity — this cross-view difference is one of the most reliable methods for detecting process hiding in memory forensics.

Task 3 — Malicious Kernel Module

With strong evidence of a rootkit, the next task asks us to identify the name of the malicious kernel module. Kernel modules (also called Loadable Kernel Modules, or LKMs) are the primary mechanism rootkits use to inject code into the Linux kernel. Once loaded, a kernel module runs with full kernel privileges and can hook system calls, manipulate process lists, hide files, and intercept network traffic.

The linux_check_modules Volatility plugin is specifically designed to detect rootkits that unlist themselves from the kernel module list. It works by comparing two sources of module information: the kernel's linked list of loaded modules (which can be tampered with) and the sysfs filesystem (which is harder to hide from). If a module appears in sysfs but not in the module list, it has been deliberately unlisted — a classic rootkit hiding technique.

$ volatility -f memory.img --profile=LinuxUbuntu_5_3_0_70-genericx64 linux_check_modules

The plugin output reveals a suspicious module named nfnetlink. At first glance, this name might appear legitimate — nfnetlink is a real Netfilter module responsible for Netfilter/netlink interface communication. However, the legitimate nfnetlink module is a core part of the Netfilter subsystem and should be properly listed in both the kernel module list and sysfs. The fact that this module is detected by linux_check_modules as being unlisted strongly suggests that the attacker has loaded a malicious kernel module that masquerades under the name of a legitimate Netfilter component, then unlisted it to avoid detection.

Task 4 — Module Load Timestamp

Identifying when the malicious kernel module was loaded is crucial for building a forensic timeline. The module load timestamp helps correlate the rootkit deployment with other events in the attack chain, such as the initial compromise, privilege escalation, or lateral movement. To find this information, we need to examine system logs that were present in memory at the time of the capture.

Extracting Syslog from Memory

The linux_enumerate_files plugin lists all files that the kernel had open or cached at the time of the memory capture. This includes log files, which often contain valuable forensic evidence. We search through the enumerated files for the syslog file, which records kernel events including module loading.

# Enumerate all files cached in memory
volatility -f memory.img --profile=LinuxUbuntu_5_3_0_70-genericx64 \
  linux_enumerate_files > files.txt

# Search for syslog
grep -i syslog files.txt

Once we locate the syslog file entry, we note its inode number and offset. We can then use the linux_find_file plugin to recover the actual file contents from the memory image. This is a powerful technique that allows us to extract files that existed on the filesystem at the time of the memory capture, even if the original disk is not available.

# Recover the syslog file using its inode offset
volatility -f memory.img --profile=LinuxUbuntu_5_3_0_70-genericx64 \
  linux_find_file -i 0xffff88007a1c2d40 -O syslog_dump.txt

Searching through the recovered syslog reveals the exact moment the malicious nfnetlink kernel module was loaded. The kernel logs a message whenever a module is inserted via insmod or modprobe, including the module name and timestamp. This gives us a precise point in time to correlate with other forensic artifacts.

Task 5 — Module File Location

Now we need to find where the malicious kernel module file is stored on the filesystem. Legitimate Netfilter modules are typically located under /lib/modules/<kernel-version>/kernel/net/netfilter/, which is the standard path for netfilter-related kernel modules in Linux. If the attacker placed their malicious module in an unusual location, this would be a clear indicator of tampering.

By examining the filesystem structure through the linux_enumerate_files output, we can locate all files with nfnetlink in their path. The legitimate nfnetlink module resides in its expected location under the net/netfilter/ subdirectory. However, we also discover an anomalous copy of the module file located in the drivers directory, which is an unusual location for a netfilter module. Netfilter modules belong in the net/ subtree, not in drivers/. This misplaced file is the malicious kernel module.

Task 6 — MD5 Hash of the Module

With the file path identified, we can recover the actual kernel module binary from the memory dump and compute its cryptographic hash. The linux_find_file plugin allows us to extract the file contents from memory, just as we did with the syslog. Once extracted, we can run standard file analysis tools against it, including hashing algorithms and string extraction.

# Recover the malicious kernel module file
volatility -f memory.img --profile=LinuxUbuntu_5_3_0_70-genericx64 \
  linux_find_file -i 0xffff88007b3f4e20 -O nfnetlink_malicious.ko

# Compute the MD5 hash
md5sum nfnetlink_malicious.ko

The MD5 hash serves as a unique fingerprint for this specific variant of the rootkit. It can be used to search threat intelligence databases, cross-reference with other incidents, or verify whether the same malware has been deployed elsewhere in the environment. The hash also provides integrity verification if the extracted file needs to be shared with other analysts or submitted to malware repositories for further analysis.

Task 8 — Author Misspelling

Running the strings command on the extracted kernel module file immediately reveals suspicious content. Kernel modules contain metadata including the module author, description, and license, which are set via MODULE_AUTHOR, MODULE_DESCRIPTION, and MODULE_LICENSE macros respectively. When an attacker creates a rootkit that masquerades as a legitimate module, they often copy this metadata from the original module — but subtle mistakes can give them away.

# Extract readable strings from the malicious module
strings nfnetlink_malicious.ko | grep -i author
author=Harald Welte <laforge@netfiter.org>

Comparing this with the legitimate nfnetlink module reveals the critical difference. The original module's author field contains the email laforge@netfilter.org, but the malicious module has the email laforge@netfiter.org — the letter "i" is missing from "netfilter", resulting in "netfiter". This is a subtle but telling mistake that the attacker made when copying the module metadata. It demonstrates that even sophisticated attackers can introduce small errors when replicating legitimate module information, and these errors become valuable forensic artifacts during investigation.

Task 9 — Initialization Function

Every Linux kernel module must define an initialization function that is called when the module is loaded via insmod or modprobe. This function is registered using the module_init() macro and serves as the entry point for the module's code. For a rootkit, the init function is where the malicious setup occurs — hooking system calls, creating hidden communication channels, and establishing persistence mechanisms.

By examining the strings in the extracted module, we can identify the initialization function name. The strings output includes symbol names and function identifiers that are embedded in the ELF binary. The init function name reveals what the rootkit calls its entry point, which can provide insights into the attacker's code structure and intent.

# Look for the init function name in module strings
strings nfnetlink_malicious.ko | grep -i init
nfnetlink_init

The initialization function is named nfnetlink_init, which further demonstrates the attacker's attempt to maintain the illusion of legitimacy. By using a name that closely follows kernel module naming conventions (<modulename>_init), the rootkit blends in with the kernel's module loading infrastructure. However, the function's actual behavior — installing hooks and hiding processes — is far from legitimate.

Task 10 — Last Hooked System Call

A critical aspect of rootkit analysis is understanding which system calls the rootkit has hooked. System call hooking is the primary mechanism by which rootkits intercept and manipulate the operating system's behavior. By replacing legitimate system call handlers with custom functions, the rootkit can filter results (hiding processes, files, or network connections), log sensitive data (like passwords), or modify behavior (elevating privileges).

The strings output from the extracted module reveals function names that correspond to system call handlers. Of particular interest is the last entry in what appears to be a system call table — this represents the final system call that the rootkit hooks, which is often the most critical one for understanding the rootkit's primary functionality.

# Extract syscall-related strings
strings nfnetlink_malicious.ko | grep -i sys
__x64_sys_kill

The presence of __x64_sys_kill is highly significant. The kill system call is used to send signals to processes, and by hooking it, the rootkit can intercept signal delivery and implement custom behavior. This is the key to understanding how the rootkit hides processes: instead of simply unlinking processes from the task list (which is what we observed earlier), the rootkit uses the kill system call as a control interface. When a specific signal number is sent to a target PID, the hooked sys_kill handler adds that PID to the hidden process list rather than actually delivering the signal.

Task 11 — Process Hiding via Signal 64

The final and most revealing task asks us to identify the specific signal number that the rootkit uses to hide processes. This ties together everything we have learned: the rootkit hooks the __x64_sys_kill system call, and within the hooked handler, it checks whether the incoming signal matches a specific value. If it does, the target PID is added to a global variable (often called global_PID) that stores the process ID to be hidden, and the signal is never actually delivered to the process.

Understanding Linux Real-Time Signals

Linux signals are divided into two categories. Standard signals (1 through 31) have predefined meanings: SIGKILL (9) terminates a process, SIGTERM (15) requests termination, SIGSTOP (19) pauses a process, and so on. These are the signals that administrators and developers use daily. However, the POSIX standard also defines real-time signals (32 through 64), which are reserved for application-defined purposes. Unlike standard signals, real-time signals have no predefined semantics — their meaning is entirely determined by the application (or in this case, the rootkit) that handles them.

/* Simplified rootkit sys_kill hook logic */
static int global_pid = 0;

asmlinkage long hooked_sys_kill(pid_t pid, int sig) {
    if (sig == 64) {
        /* Signal 64 = hide this PID */
        global_pid = pid;
        return 0;  /* Swallow the signal */
    }
    /* Otherwise, pass through to original sys_kill */
    return orig_sys_kill(pid, sig);
}

The rootkit uses signal 64 as its control channel. When the attacker sends signal 64 to any process ID using kill -64 <PID>, the hooked sys_kill handler intercepts the call, stores the target PID in the global_PID variable, and returns success without actually delivering the signal. The rootkit's process enumeration hooks then filter out any process whose PID matches global_PID, effectively making it invisible to all userland process listing tools.

This approach is elegant from the attacker's perspective because it provides a simple, command-line interface for hiding and unhiding processes dynamically. The attacker can hide any process with a single command, and the change takes effect immediately across the entire system. No special tools or custom binaries are required — the standard kill command is sufficient to control the rootkit's behavior.

Investigation Summary

APTNightmare2 takes us through a complete rootkit investigation using Linux memory forensics. Starting from a raw memory dump, we methodically uncovered the attacker's infrastructure and techniques using Volatility's specialized plugins. The investigation followed a logical progression: identifying the reverse shell connection, discovering hidden processes through cross-view detection, pinpointing the malicious kernel module, extracting forensic artifacts from memory, and ultimately understanding the rootkit's internal mechanism for process hiding.

The rootkit demonstrates several sophisticated techniques that make it a formidable forensic challenge. Its use of name mimicry (masquerading as the legitimate nfnetlink module), strategic file placement in an unusual directory, process unlinking from the kernel task list, and the elegant signal-based control interface via __x64_sys_kill hook all represent real-world rootkit design patterns. However, each technique leaves traces in memory that can be uncovered through careful forensic analysis — demonstrating that no matter how stealthy a rootkit attempts to be, memory forensics provides a powerful lens for revealing the truth.