Testing SIEM/EDR Solutions using Simulation Tool — Caldera

Testing Endpoint Detection Solutions using Caldera

4 min readJun 7, 2021

What is Caldera?

Caldera is a cyber security framework designed to easily run autonomous breach simulation exercises. It is used to automate adversary emulation and incident response. The core framework consists of an asynchronous command-and-control (C2) server with a REST API and a web interface. Caldera also supports plugins which expands the core framework capabilities and provides additional functionalities. Its main use is to test endpoint solutions.

Here, I used Caldera to test a SIEM’s (Splunk) integration with the OS. The main goal is to simulate adversarial behavior and check with of these activities are picked up by the Monitoring SIEM agent. A more common solution is Red Teaming which is time consuming and expensive, whereas using a simulation tool like Caldera can save time and money but these tools can only be used to conduct initial testing of SIEM’s/EDR’s deployment in the environment.

Lab Setup: A Kali Linux VM acting as a C2C Server using Caldera and a Windows 10 VM as a victim with Caldera RAT agent and Splunk’s Monitoring agent (Splunk Universal Forwarder). Both the machines are connected via an internal network.

Installing Caldera

Installing Caldera is a very straightforward process and should take around 10 minutes or less. Start by cloning the repository in the desired folder.

git clone https://github.com/mitre/caldera.git — recursive — branch 3.0.0

Navigate to the caldera folder and install all the requirements

pip3 install -r requirements.txt

From within the folder itself, start the Caldera Server GUI

python3 server.py –insecure

Now, login to the Caldera Server GUI on http://localhost:8888 using the credentials, red:admin.

Establish Caldera agents

Caldera agents are Remote Access Tools (RATs), allowing to execute benign commands which simulate adversarial behavior. Using the GUI navigate to agents — > Add agents.

Now, “*Select Click here to deploy an agent*” to select which agent to add.

54ndc47 (SandCat) setup code for Windows Powershell

Now, select 54ndc47 (SandCat) as your agent and select the OS, add the host system’s IP address in app.contact.http and append port 8888 at the end (default listening port for Caldera). Then head over to the Victim machine/Endpoint. For windows, paste the command in powershell (admin privileges are optional) and for Ubuntu/Debian paste the command in the terminal.

Launch an Adversarial Operation

Now, we have everything ready to carry out our Adversarial attacks using Caldera. It has many pre-defined Adversarial Plans which are just to get us understand the system. The best ones to start are “The Discovery” or “The Hunter”.

From the menu navigate to “Operations”, toggle the switch to “Add”. Type a name for the operation. Under the basic operations, select the adversary plan and change “Keep open forever” to “Auto close operation”. Press Start to begin you attack.

Results: Detections in SIEM!

Upon running multiple operations in Caldera these are the detections in Splunk.

Splunk Detections based on Caldera Simulation

In conclusion, Caldera can be used to test endpoint security solutions and assess a network’s security posture against the common post-compromise adversarial techniques. While the use of Caldera can reduce resources needed for assessments and allow red teams to focus on sophisticated solutions to harder problems. It will also allow organizations to more rapidly tune behavioral-based intrusion detection systems as they are deployed. Caldera is more of a complementary to other methods of security assessment, but its insights in threat analysis and behavior can help blue teams anticipate certain adversarial behaviors and make necessary decisions.