Linux Host Compromise Playbook

Published on 5th September, 2023

By Sivaraju Kuraku

Introduction

In today’s digital landscape, ensuring the security and integrity of Linux hosts is paramount. As Linux servers are frequently targeted by cyber attackers, having a comprehensive playbook to identify, investigate, and remediate suspicious activities is crucial for system administrators and security professionals. This playbook provides detailed steps and commands to help in examining various aspects of a Linux system, such as file paths, processes, services, cron jobs, network connections, and persistence mechanisms. By following these guidelines, you can effectively detect and mitigate potential compromises on your Linux hosts, thereby maintaining a secure and resilient infrastructure.

Common Places to Search for Suspicious Activity

1. /var/spool/cron – Location of user crontabs
2. /etc/cron/.d – Other cron jobs
3. /home – User home directory
4. /var/log – Local log files
5. /tmp – Most common place for malware

File Paths

Investigation Commands and Purposes

Paths: /tmp, /var/tmp, /dev/shm
Command: find /tmp /var/tmp /dev/shm -type f
Purpose: Temp directories are commonly used by attackers to store malware or scripts temporarily. Investigating these paths can uncover hidden malicious files.

Paths: /home/[username]/.ssh/, /home/[username]/.bashrc, /home/[username]/.bash_profile
Command: grep -iR ‘base64\|wget\|curl’ /home/[username]/
Purpose: Searches for base64 encoding and download commands in user profiles and SSH configurations, which might indicate backdoor setup or persistence mechanisms.

Paths: Web root directories, e.g., /var/www/html, /usr/share/nginx/html
Command: find /var/www/html -name “*.php” -exec grep -l ‘eval\|base64_decode’ {} \;
Purpose: Identifies PHP files containing eval or base64_decode functions, often used in web shells or malicious scripts.

Paths: /bin, /usr/bin, /sbin, /usr/sbin
Command: find / -perm -4000 -exec ls -ldb {} \;
Purpose: Finds all SUID (Set User ID upon execution) files. Malicious binaries might be placed here to execute with elevated privileges.

Paths: /etc/passwd, /etc/shadow, /etc/cron*, /etc/systemd/system
Command: auditctl -w /etc/passwd -p wa -k passwd_changes
Purpose: Monitors changes to critical configuration files, alerting on unauthorized modifications.

Paths: /etc/crontab, /var/spool/cron/crontabs/, /etc/cron.d/
Command: grep -Ri ‘base64\|curl\|wget’ /etc/cron* /var/spool/cron/crontabs/ /etc/cron.d/
Purpose: Identifies suspicious commands in cron jobs that could be used for executing malicious activities on a schedule.

Paths: /home/[username]/.ssh/authorized_keys
Command: cat /home/[username]/.ssh/authorized_keys | grep ‘ssh-rsa’
Purpose: Checks for unauthorized SSH keys that may grant attackers persistent remote access.

Paths: Suspicious Hidden Files or directories starting with .
Command: find / -name “.*” -type f
Purpose: Finds hidden files which may be used to store malicious scripts or data.

Paths: /var/log/, specifically auth.log, syslog, or messages showing gaps or edits
Command: ls -lt /var/log/ | head
Purpose: Checks for recent modifications in log files that could indicate tampering to hide malicious activities.

Remediation Commands

• Command: rm -f /suspicious/path/malicious_file – Deletes identified malicious files. Use cautiously to avoid removing legitimate system files.
• Command: chmod 644 /suspicious/file – Resets permissions on files that may have been modified to prevent unauthorized access.
• Command: Review and harden configuration files, e.g., vi /etc/ssh/sshd_config – Ensures services are configured securely, minimizing the attack surface. Focus on disabling root login, enforcing key-based authentication for SSH, and applying the principle of least privilege.
• Command: echo ” > /home/[username]/.ssh/authorized_keys – Clears all authorized keys to remove unauthorized access. Ensure legitimate keys are backed up and re-added carefully.

Investigating and Remediating Linux Processes

• Command: ps aux – Displays all running processes. It is useful for spotting unusual or unknown processes that could signify a compromise.
• Command: top or htop – Provides a dynamic, real-time view of running processes. It helps in identifying processes that are using excessive system resources, which may indicate malicious activity.
• Command: pstree -p – Shows running processes as a tree, making it easier to understand the parent-child relationships between processes. This can help track the origin of suspicious processes.
• Command: lsof -p [PID] – Lists all files opened by a specific process (indicated by its PID). This is crucial for identifying what resources a suspicious process is accessing.
• Remediation Command: kill -9 [PID] – Immediately stops a specific process identified as potentially malicious. The -9 option ensures a forceful termination.

Investigating and Remediating Linux Services

• Command: systemctl list-units –type=service –state=running – Lists all currently active services. This command helps identify unexpected or unauthorized services that may be running.
• Command: systemctl status [service_name] – Provides detailed information about a specific service, including its current status, recent logs, and whether it’s enabled to start at boot. It is useful for diagnosing issues with known services or investigating suspicious ones.
• Command: journalctl -u [service_name] – Displays the systemd journal logs for a specific service. Logs can reveal errors, unauthorized access attempts, or other suspicious activities associated with the service.
• Command: cat /etc/systemd/system/[service_name].service – Displays the contents of the service’s unit file, which defines how the service starts and operates. Reviewing this file can help identify malicious configurations or modifications.
• Remediation Command: systemctl stop [service_name] – Stops a service that might be running a malicious process. This is a softer approach than killing a process and is useful for services that will be investigated further.
• Remediation Command: systemctl disable [service_name] – Prevents the service from automatically starting on system boot, reducing the risk of the persistence of malicious activities.
• Remediation Command: systemctl mask [service_name] – Prevents the service from being started manually or automatically. Masking is a stronger version of disable, ensuring the service cannot be activated without being unmasked.
• Remediation Command: rm /etc/systemd/system/[service_name].service and then systemctl daemon-reload – Deletes the service’s unit file and reloads the system to apply changes. Use this to remove unauthorized or malicious services permanently.
• Restore Command: sudo systemctl revert [service_name] – Restores the original system service file if it has been modified. This can be useful if a legitimate service’s configuration is altered for malicious purposes.

Investigating and Remediating Cron Jobs and Scheduled Tasks

• Command: for user in $(cut -f1 -d: /etc/passwd); do echo “Cron jobs for $user:”; crontab -u $user -l; done – Enumerates all cron jobs across all user accounts, including system users, which helps in identifying unexpected or malicious entries.
• Command: ls -al /etc/cron* – Lists content in system-wide cron directories (cron.daily, cron.hourly, cron.monthly, cron.weekly, and cron.d). Reviewing these directories is crucial for spotting unauthorized scheduled tasks.
• Command: cat /etc/anacrontab – Displays tasks scheduled with Anacron, which, unlike Cron, can run commands not strictly tied to the system’s clock. It’s vital for systems that don’t run 24/7.
• Command: systemctl list-timers –all – Lists all active systemd timers, an alternative to cron jobs used in newer distributions. This command uncovers scheduled tasks managed by systemd.
• Remediation Command: crontab -u [username] -e – Opens the crontab editor for a specific user, allowing the removal of unauthorized or malicious cron jobs.
• Remediation Command: chmod -x /etc/cron.daily/[job_name] – Removes execution permissions from a system-wide cron job script, effectively disabling it without deletion.
• Remediation Command: rm /etc/cron.d/malicious_cron – Completely removes a cron job file from the system-wide cron directory, used when a cron job is identified as malicious.
• Remediation Command: systemctl stop [timer_name].timer && systemctl disable [timer_name].timer – Stops and disables a systemd timer, preventing the scheduled task from running again.

Investigating and Remediating Network Connections

• Command: ss -tunapl or netstat -tunapl – Lists all active network connections (TCP/UDP), along with the processes responsible for them. This can help identify unauthorized connections or processes communicating with suspicious external IP addresses.
• Command: lsof -i – Lists open network connections and the associated processes. Useful for understanding which processes are communicating over the network.
• Command: iptables -L -n -v – Displays the current firewall rules, which can reveal unauthorized or malicious rules that allow or block certain traffic.
• Command: tcpdump -i eth0 – Captures network traffic on a specific interface. Analyzing this traffic can help detect unusual patterns or data exfiltration attempts.
• Remediation Command: iptables -A INPUT -s [suspicious_IP] -j DROP – Blocks incoming traffic from a suspicious IP address, helping to mitigate ongoing attacks.
• Remediation Command: iptables -D INPUT -s [suspicious_IP] -j ACCEPT – Removes an existing rule that allows traffic from a suspicious IP address, tightening network security.
• Remediation Command: kill -9 $(lsof -ti:[port]) – Forcefully terminates all processes using a specific network port, useful when a port is identified as being used for malicious communication.
• Remediation Command: ss -K dst [suspicious_IP] – Terminates all network connections to a specific IP address, helping to disrupt active malicious communications.
• Remediation Command: tcpkill host [suspicious_IP] – A tool that allows for the targeted termination of network connections, which is useful for quickly stopping communication with known malicious IPs.

Conclusion

By following the outlined procedures and utilizing the provided commands, you can systematically investigate and remediate suspicious activities on your Linux hosts. Regularly updating and practicing these steps ensures that you stay prepared to address potential threats effectively. Remember, maintaining a secure environment is an ongoing process that requires vigilance, regular updates, and continuous improvement of your security practices. By implementing this playbook, you take a significant step towards safeguarding your systems against potential compromises.