ReaQta has successfully completed the MITRE evaluation, showing ReaQta-Hive’s capabilities of providing complete coverage of sophisticated attacks, with no human intervention and top-quality alerts. Let us start off with understanding what MITRE evaluation is all about and then discuss how ReaQta performed during the test.

What is MITRE ATT&CK Evaluation

MITRE ATT&CK has defined a set of stages during a cyberattack and evaluates solutions on their ability to detect threats in each of these. Each of the listed stages represents a “tactic” along the kill-chain:

  1. Initial Access
  2. Execution
  3. Persistence
  4. Privilege Escalation
  5. Defense Evasion
  6. Credential Access
  7. Discovery
  8. Lateral Movement
  9. Collection
  10. Exfiltration
  11. Command and Control

MITRE has a set of identified techniques, each of which belongs to a tactic group, based on the threat actor that they select for the evaluation. MITRE chose APT29 for this round of evaluation.

How Does the MITRE Evaluation Help Organisations

The evaluation does not score or grade solutions and it is meant to help organisations identify the most suitable solution that meets their specific security challenges. Organisations do need to note that the evaluation takes place in isolated environments and have limitations. There are times when certain features of a solution are disabled, as they do not support that particular lab infrastructure, like in case of ReaQta, the NanoOS, our live hypervisor used to detect high-level malicious behaviours, could not be used. Nonetheless the platform performed well, even without its core component.

No Manual (MSSP) Detections

Before starting the evaluation ReaQta decided to participate without MSSP, that is without any human interaction during the attack. MITRE is a technology evaluation framework and we felt it would be unfair to introduce humans in the loop. On top of that the contribution of MSSP detections heavily biases the evaluation. The SOC team knows that an attack is happening and they know exactly where and how.

We felt that the MSSP approach wouldn’t have provided our customers with a fair assessment of the technology. MITRE has been very receptive to feedback and starting from Round 3 all companies will be evaluated without humans in the loop

We think MSSPs add great value, customers should be free to choose between MSSP and stand-alone deployments, but we don’t think MSSPs belong to the evaluation framework as it’s easy to skew the results in one’s favour.

As we can see from the graph below, the amount of detections performed by humans had a huge impact on generated detections, in several instances more than 50% of detections – and up to 73% – were created manually. Only 6 companies decided to participate without humans in the loop.

Manual detections generated by each vendor throughout the evaluation process.
Manual detections generated by each vendor.

The configuration used by ReaQta during the evaluation can be found here.

MITRE Evaluation Round 2 – APT29

Vendors were tested on their ability to detect the tactics and techniques used by APT29 (also known as The Dukes, Cozy Bear and CozyDuke), a sophisticated nation-state adversary known for their stealthy approach. APT29 is widely-known for being behind notable attacks: the Pentagon in 2015, the Democratic National Committee in 2016, the Norwegian and Dutch governments in 2017.

The change from the previous round was important: APT3 (Round 1) is a noisy threat actor, adopting a variety of tools with much less regard to maintaining a low profile. APT29 on the other hand is extremely stealthy, operating with a very low-profile and leveraging heavily on LOLbins and file-less malware.

ReaQta Evaluation Results

The attack unfolded over 2 days in which the attackers gradually moved deeper into the network after obtaining initial access. The vast majority of operations were carried out using powershell, as opposed to custom tools and malware, in order to maintain a low detection profile. The evaluation goal is to show how tested solutions respond to the attack and what kind of visibility is provided along the entire kill-chain.

ReaQta-Hive automated detection coverage compared to the average

Visibility Across the ATT&CK Kill-Chain

As is evident from the summary of the evaluation results above, ReaQta-Hive platform provided complete visibility across the entire kill-chain. ReaQta-Hive detected 90% of the Tactics and Techniques tested, proving its ability to respond and remediate threats at every stage of the attack.

“Since there is also a need to detect and respond to unknown, fileless and advanced persistent threats (including those associated with state sponsored attackers), there must also be an assumption that simply trying to prevent all exploits is unrealistic.”

Gartner, Market Guide for Endpoint Detection and Response Solutions

Right Alerts at Critical Stages

The platform detected and generated alerts right from Execution, Persistence, Privilege Escalation and Defense Evasion stages, enabling the security team to track APT29 and their actions as the attack unfolded. The platform alerts were consistent during the later Kill-chain stages: Lateral Movement, Collection, Exfiltration and Command and Control, showing ReaQta-Hive’s ability to respond and limit damages also in the late stages of a cyber attack.

Actionability is the product of Alert Efficiency and Alert Quality […] efficiency of alerts (not too many) and the quality of the alerts (how well they help you understand the story) are both related and critical to understanding how “actionable” a particular alert is going to be.

Forrester

ReaQta-Hive shows one of the world’s top Actionability rates, even when compared against vendors relying on Manual (MSSP) detections. The chart below uses data extracted by Forrester’s analysts.

Alerts actionability rates
Actionability rates (data includes Manual detections for vendors relying on MSSPs)

The Actionability rate highlights the platform’s capability to reduce noise by reducing the amount of alerts generated. The platform captures all tactics and techniques in a few correlated alerts, as compared to one alert per tactic and technique, which would amount to an unmanageable number of alerts for the SOC teams to examine and respond. 

Providing high-fidelity and comprehensive alerts is the criteria that sets a good platform aside from noise generators. With the amount of visibility provided by ReaQta-Hive it is necessary to filter data, correlate it and generate the smallest amount of alerts possible, each containing the largest amount of related information. This is the purpose of our A.I. engines: collect, correlate and summarise the telemetry. Alerts Quality is also confirmed by Forrester’s analysis in the chart below.

Alerts Quality (data includes Manual detections for vendors relying on MSSPs)

Once again, ReaQta-Hive provides high-quality alerts without human intervention, while both the first and third vendors relied on manual analysis during the evaluation.

The graph below shows how ReaQta-Hive behaves compared to other solutions when manual detections are removed. Each bar represents the amount of incident-related information captured under each generated alert. Our engines have captured the largest amount of information, that translates to a sizeable work-load reduction in real environments.

Percentage of attack coverage provided per alert
Percentage of attack coverage provided per alert

To provide an example related to the evaluation, in the image below we can see how an entire stage of the attack has been captured within a single alert. ReaQta-Hive has correlated all the information into an easily comprehensible storyline, thereby providing to a SOC team all the information for timely triage. No human interaction was required and the attack is cleanly explained, and its risk assessed, without requiring any manual activity.

MITRE Evaluation Storyline
ReaQta-Hive correlated Storyline during MITRE evaluation

The ability to provide a unified incident resolution workflow is critical to reduce alert fatigue. It allows analysts to understand and study an active attacker, without being distracted by hundreds of alerts being generated with no direct correlation with the original incident. 

During the entire course of the evaluation, ReaQta-Hive generated just 25 alerts and correctly gathered all the information required to track the attackers within each one of them, instead of creating 158 alerts (one per technique tested), which would have been much harder to handle during a real analysis. ReaQta-Hive approach reduced the alert fatigue by 85% while preserving complete visibility over the entire attack.

ReaQta-Hive is specifically designed to generate the minimal amount of alerts per incident, allowing for a smooth and uninterrupted analysis experience. The ability to maintain everything in a single view helps analysts to respond faster, without requiring jumps to different screen-views, in order to have a complete understanding of the events.

ATT&CK Tactics and Techniques  with Complete Visibility

The platform was able to  maintain correlation between actions at all stages of the ATT&CK kill-chain. Correlating events automatically reduces the time needed to piece together different actions run by the attackers and ultimately it reduces the response time in case of real attacks.

MITRE Techniques Visibility
List of detected Tactics and Techniques

A closer look at the detection of APT29 tactics and techniques, ReaQta-Hive provided visibility right from the early stages of the kill-chain to the more sophisticated stages which are often harder to detect. What is noteworthy here is the platform’s ability to uniformly detect threats at every stage, thereby providing opportunities for response and remediation at every stage.

ReaQta-Hive showed one of the world’s best telemetries, combined with an impressive A.I. engine capable of condensing information and assessing risk, it will prove a powerful tool in the hands of any SOC or team that wants to spend time threat hunting instead of managing alerts constantly.

Amount of telemetry provided by ReaQta-Hive

The Way Ahead

ReaQta’s AI-powered platform was designed to equip security teams with advanced detection and rapid response capabilities, minimizing human intervention, simplifying the entire cybersecurity process and ensuring business continuity for organisations of all sizes. 

We highly value the feedback that the community gives us and MITRE evaluation was a step forward in this direction. This evaluation has validated ReaQta’s approach to the detection of sophisticated threat actors. ReaQta will continue to participate in independent third party testing in the future.

ReaQta appreciates and applauds the work of MITRE in helping organisations from making informed decisions with these evaluations.