Incident handling with Splunk Tryhackme Walkthrough

Learn to use Splunk for incident handling through interactive scenarios

This write-up covers the Incident handling with Splunk Room on TryHackMe.
Learning Path (s): Cyber Defense, SOC Level 1
Module: Security Information and Event Management
Skill/Tools: SIEM Tools, Splunk
Setting up: Connect to the Start the Tryhackme VM or deploy the machine locally via OpenVPN

Investigating a cyber attack that has defaced the victim’s website, “imreallynotbatman.com”.

1 & 2: Incident Handling — Life Cycle

💡An incident from a security perspective is “Any event or action that has a negative consequence on the security of a user, computer, or organization is considered a security incident.”

Below are a few of the events that would negatively affect the environment when they occurred:

• Crashing the system
• Execution of an unwanted program
• Access to sensitive information from an unauthorized user
• A Website being defaced by the attacker
• The use of USB devices, when there is a restriction in usage, is against the company’s policy

Learning Objective

• Learn how to leverage OSINT sites during an investigation
• How to map Attacker’s activities to Cyber Kill Chain Phases
• How to utilize effective Splunk searches to investigate logs
• Understand the importance of host-centric and network-centric log sources

Incident Handling Life Cycle

As a SOC analyst, any incident should be handled using the Incident Handling Lifecycle. As an Incident handler or SOC Analyst, it is important to understand the attackers’ tactics, techniques, and procedures (TTPs) to better defend against attacks. The Incident Handling Life Cycle consists of four phases:

1. The Preparation phase involves documenting requirements, defining policies, and incorporating security controls such as EDR/SIEM, IDS and IPS, as well as hiring and training staff.
2. The Detection and Analysis phase covers detecting incidents and analyzing them, including investigating alerts from security controls and hunting for unknown threats.
3. The Containment, Eradication, and Recovery phases involve preventing the incident from spreading and securing the network by isolating infected hosts and clearing the network of infection traces.
4. The Post-Incident Activity/Lessons Learned phase involves identifying weaknesses in the organization’s security posture and improving it to prevent future attacks, including adding detection rules and training staff if necessary.

3: Incident Handling: Scenario

💡We will investigate a cyberattack where the attacker defaced an organization’s website. Our task is to map the attacker’s activities into all seven phases of the Cyber-Kill Chain Phases.

To understand how an incident happened, we also need to think from an attacker’s point of view, using The Cyber Kill Chain Phases as a guide.

ㅤ

Scenario

A Big corporate organization Wayne Enterprises has recently faced a cyber-attack where the attackers broke into their network, found their way to their web server, and have successfully defaced their website http://www.imreallynotbatman.com. Their website is now showing the trademark of the attackers with the message YOUR SITE HAS BEEN DEFACED, as shown below.

They have requested us to join them as a Security Analyst and help them investigate this cyber attack and find the root cause and all the attackers’ activities within their network.

The good thing is that they have Splunk already in place, so we have all the event logs related to the attacker’s activities captured. We need to explore the records to find out how the attack got into their network and what actions they performed.

trademark of the attackers with the message

This Investigation comes under the Detection and Analysis phase.

We will use Splunk as our SIEM solution during the investigation, which ingests logs from the webserver/firewall/Suricata/Sysmon.

In the data summary tab, we can explore the log sources, showing visibility into both network-centric and host-centric activities.

All the event logs we are going to investigate are present in index=botsv1. Some of the interesting log sources that will help us in our investigation are available. Let’s connect to the lab and start investigating.

4: Reconnaissance Phase

💡Reconnaissance is an attempt to discover and collect information about a target. It could be knowledge about the system in use, the web application, employees or location, etc.

We will start our analysis by examining any reconnaissance attempts against the webserver imreallynotbatman.com. We need to look at the network traffic log sources to identify the IP address attempting to perform reconnaissance activity on our web server.

1. We can begin by searching for the domain in the search head and see which log source includes the traces of our domain.
   Search Query: index=botsv1 imreallynotbatman.com
2. We’ll then look at the log source stream:http, which contains the http traffic logs, and examine the src_ip field from the left panel. Src_ip field contains the source IP address it finds in the logs.
   Search Query: index=botsv1 imreallynotbatman.com sourcetype=stream:http

We found two IPs in the src_ip field 40.80.148.42 and 23.22.63.114.

1. To further confirm our suspicion about the IP address 40.80.148.42, click on the IP and examine the logs. We can look at the interesting fields like User-Agent, Post request, URIs, etc., to see what kind of traffic is coming from this particular IP.
2. To validate the scanning attempt, we’ll look into the Suricata logs and search for the fields that contained the signature alert information.
   Search Query: index=botsv1 imreallynotbatman.com src=40.80.148.42 sourcetype=suricata
In the right panel, we could not find the field of our interest, so we clicked on more fields and searched for the fields that contained the signature alerts information, which is an important point to note.

5: Exploitation Phase

💡In this task, we will look at the potential exploitation attempt from the attacker against our web server and see if the attacker was successful in exploiting it or not.

Count: Let’s use the following search query to see the number of counts by each source IP against the webserver.
Search Query:index=botsv1 imreallynotbatman.com sourcetype=stream* | stats count(src_ip) as Requests by src_ip | sort - Requests

We can also create different visualization to show the result. Click on Visualization → Select Visualization as shown below.

We narrowed down the result to show requests sent to our web server, which has the IP 192.168.250.70, and observed most of the requests coming through the POST request.
Search Query: index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70"
The result in the src_ip field shows three IP addresses (1 local IP and two remote IPs) that originated the HTTP traffic towards our webserver.

We will then narrow down on the field http_method=POST
Search Query: index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" http_method=POST

The term Joomla is associated with the webserver found in a couple of fields like uri, uri_path, http_referrer, etc. This means our webserver is using Joomla CMS (Content Management Service) in the backend.

A little search on the internet for the admin login page of the Joomla CMS will show as -> /joomla/administrator/index.php

We can narrow down our search to see the requests sent to the login portal using this information.
Search query: index=botsv1 imreallynotbatman.com sourcetype=stream:http dest_ip="192.168.250.70" uri="/joomla/administrator/index.php"

We suspected that the attacker may have tried multiple credentials in an attempt to gain access to the admin panel, so we dug deep into the values contained within the form_data field and found that the attacker from the IP 23.22.63.114 was trying to guess the password by brute-forcing and attempting numerous passwords.
Search Query: index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" http_method=POST uri="/joomla/administrator/index.php" | table _time uri src_ip dest_ip form_data

Extracting Username and Passwd Fields using Regex: We can display only the logs that contain the username and passwd values in the form_data field by adding form_data=*username*passwd* in the above search.
Search Query: index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" http_method=POST uri="/joomla/administrator/index.php" form_data=*username*passwd* | table _time uri src_ip dest_ip form_data

It’s time to use Regex (regular expressions) to extract all the password values found against the field passwd in the logs. To do so, Splunk has a function called rex.

Now, let’s use Regex. rex field=form_data "passwd=(?<creds>\w+)" To extract the passwd values only.
Search Query:index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" http_method=POST form_data=*username*passwd* | rex field=form_data "passwd=(?<creds>\w+)" | table src_ip creds

We extracted all the password values found against the field passwd in the logs using Regex and found that one request came from a Mozilla browser while others were from a Python script used for automation.

Let’s create a table to display key fields and values by appending -> | table _time src_ip uri http_user_agent creds in the search query
Search Query: index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" http_method=POST form_data=*username*passwd* | rex field=form_data "passwd=(?<creds>\w+)" |table _time src_ip uri http_user_agent creds

This result clearly shows a continuous brute-force attack attempt from an IP 23.22.63.114 and 1 password attempt batman from IP 40.80.148.42 using the Mozilla browser.

6: Installation Phase

💡This phase will investigate any payload / malicious program uploaded to the server from any attacker’s IPs and installed into the compromised server.

We would first narrow down any HTTP traffic coming into our server 192.168.250.70 containing the term “.exe.” This query may not lead to the findings, but it’s good to start from 1 extension and move ahead
Search Query: index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" *.exe

As we could not find the file name field, we looked at the missing fields and saw a field. part_filename{}.
Observing the interesting fields and values, we can see the field part_filename{} contains the two file names. an executable file 3791.exe and a PHP file agent.php

Next, we need to find if any of these files came from the IP addresses that were found to be associated with the attack earlier. Click on the file name; it will be added to the search query, then look for the field c_ip, which seems to represent the client IP.
Search Query:index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" "part_filename{}"="3791.exe"

We have found that file 3791.exe was uploaded to the server. The question that may come to our mind would be, was this file executed on the server? We need to narrow down our search query to show the logs from the host-centric log sources to answer this question.
Search Query: index=botsv1 "3791.exe"

For the evidence of execution, we can leverage sysmon and look at the EventCode=1 for program execution.
Search Query: index=botsv1 "3791.exe" sourcetype="XmlWinEventLog" EventCode=1

Looking at the output, we can clearly say that this file was executed on the compromised server. We can also look at other host-centric log sources to confirm the result.

7: Action on Objectives

💡As the website was defaced due to a successful attack by the adversary, it would be helpful to understand better what ended up on the website that caused the defacement. As an analyst, our first quest could be to figure out the traffic flow that could lead us to the answer to this question.

We will start our investigation by examining the Suricata log source and the IP addresses communicating with the webserver 192.168.250.70.
Search Query:index=botsv1 dest=192.168.250.70 sourcetype=suricata

The logs do not show any external IP communicating with the server. Let us change the flow direction to see if any communication originates from the server.
Search Query: index=botsv1 src=192.168.250.70 sourcetype=suricata

There is a large chunk of traffic going to these external IP addresses, which could be worth checking. Pivot into the destination IPs one by one to see what kind of traffic/communication is being carried out.
Search Query: index=botsv1 src=192.168.250.70 sourcetype=suricata dest_ip=23.22.63.114

The URL field shows 2 PHP files and one jpeg file. This jpeg file looks interesting. Let us change the search query and see where this jpeg file came from.
Search Query: index=botsv1 url="/poisonivy-is-coming-for-you-batman.jpeg" dest_ip="192.168.250.70" | table _time src dest_ip http.hostname url

The end result clearly shows a suspicious jpeg poisonivy-is-coming-for-you-batman.jpeg was downloaded from the attacker's host prankglassinebracket.jumpingcrab.com that defaced the site.

8: Command and Control Phase

💡The attacker uploaded the file to the server before defacing it. While doing so, the attacker used a Dynamic DNS to resolve a malicious IP. Our objective would be to find the IP that the attacker decided the DNS.

To investigate the communication to and from the adversary’s IP addresses, we will be examining the network-centric log sources mentioned above. We will first pick fortigate_utm to review the firewall logs and then move on to the other log sources.
Search Query: index=botsv1 sourcetype=fortigate_utm"poisonivy-is-coming-for-you-batman.jpeg"

Looking into the Fortinet firewall logs, we can see the src IP, destination IP, and URL. Look at the fields on the left panel and the field url contains the FQDN (Fully Qualified Domain Name).

Let us verify the answer by looking at another log source.stream:http.
Search Query: index=botsv1 sourcetype=stream:http dest_ip=23.22.63.114 "poisonivy-is-coming-for-you-batman.jpeg" src_ip=192.168.250.70

We have identified the suspicious domain as a Command and Control Server, which the attacker contacted after gaining control of the server.

Note: We can also confirm the domain by looking at the last log source stream:dns to see what DNS queries were sent from the webserver during the infection period.

9: Weaponization Phase

💡In the weaponization phase, adversaries create malware or malicious documents to gain initial access, evade detection, etc. They establish domains similar to the target domain to trick users and create a command and control server for post-exploitation communication/activity.

We found a domain associated with the attack, prankglassinebracket.jumpingcrab.com, and our first task is to find the IP address tied to the domains that may potentially be pre-staged to attack Wayne Enterprise.

Robtex: We searched for the domain on Robtex and found some domains/subdomains associated with it. We also searched for the 23.22.63.114 IP address on Robtex and found that it is associated with domains that look similar to Wayne Enterprise’s site.

Virustotal: We then searched for the IP address on Virustotal and found all domains associated with this IP that look similar to Wayne Enterprise’s site. We also found the domain associated with the attacker, www.po1s0n1vy.com, in the domain list and searched for it on virustotal.

WHOIS: Finally, we looked for whois information on whois.domaintools.com to see if we could find something valuable.

10: Delivery Phase

💡Attackers create malware and infect devices to gain initial access or evade defenses and find ways to deliver it through different means. Our task for this lesson would be to use the information we have about the adversary and use various Threat Hunting platforms and OSINT sites to find any malware linked with the adversary.

Threat Intel report suggested that this adversary group Poison lvy appears to have a secondary attack vector in case the initial compromise fails. Our objective would be to understand more about the attacker and their methodology and correlate the information found in the logs with various threat Intel sources.

OSINT sites

1. ThreatMiner
   Look for the IP 23.22.63.114 on ThreatMiner

We found three files associated with the IP, and one file with the hash value c99131e0169171935c5ac32615ed6261 seems to be malicious and of interest. Click on this MD5 hash value to see metadata and other important information about the file.

2. Search for the hash on the VirusTotal

3. Hybrid-Analysis is a site that shows the behavior analysis of any malware, including network communication, DNS requests, contacted hosts with country mapping, strings, MITRE ATT&CK mapping, malicious indicators, DLLs imports/exports, mutex information if created, file metadata, and screenshots.

Room Q&A

Note: Due to Medium’s formatting, some elements are dropped. Please visit our Notion page for the full tutorial.

Splunk

Task 4:

One suricata alert highlighted the CVE value associated with the attack attempt. What is the CVE value?

What is the CMS our web server is using?

What is the web scanner, the attacker used to perform the scanning attempts?

What is the IP address of the server imreallynotbatman.com?

Task 5

What IP address is likely attempting a brute force password attack against imreallynotbatman.com?

What was the URI which got multiple brute force attempts?

Against which username was the brute force attempt made?

What was the correct password for admin access to the content management system running imreallynotbatman.com?

• “When looking at web traffic, if an attack was successful, the Content-Length field under src_headers in Splunk can provide valuable information.”
• index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" http_method=POST form_data=*username*passwd* | rex field=form_data "passwd=(?<creds>\w+)" |table _time src_ip uri http_user_agent creds src_headers

How many unique passwords were attempted in the brute force attempt?

• index=botsv1 sourcetype=stream:http dest_ip="192.168.250.70" http_method=POST form_data=*username*passwd* | rex field=form_data "passwd=(?<creds>\w+)" |stats count(creds) as UniquePassword by src_ip

After finding the correct password, which IP did the attacker use to log in to the admin panel?

Task 6

Sysmon also collects the Hash value of the processes being created. What is the MD5 HASH of the program 3791.exe?

Looking at the logs, which user executed the program 3791.exe on the server?

Search hash on the virustotal. What other name is associated with this file 3791.exe?

Task 7

What is the name of the file that defaced the imreallynotbatman.com website ?

Fortigate Firewall ‘fortigate_utm’ detected SQL attempt from the attacker’s IP 40.80.148.42. What is the name of the rule that was triggered during the SQL Injection attempt?

Task 8

This attack used dynamic DNS to resolve to the malicious IP. What fully qualified domain name (FQDN) is associated with this attack?

Task 9

What IP address has P01s0n1vy tied to domains that are pre-staged to attack Wayne Enterprises?

Based on the data gathered from this attack and common open-source intelligence sources for domain names, what is the email address that is most likely associated with the P01s0n1vy APT group?

• Googling for the IOCs collected so far leads to https://threatcrowd.org/ip.php?ip=23.22.63.114 where we are presented with a relationship diagram involving domains, IPs, emails:

Task 10

What is the HASH of the Malware associated with the APT group?

What is the name of the Malware associated with the Poison Ivy Infrastructure?