{"id":304515,"date":"2023-09-20T11:20:10","date_gmt":"2023-09-20T18:20:10","guid":{"rendered":"https:\/\/www.paloaltonetworks.com\/blog\/?p=304515"},"modified":"2023-10-05T10:50:23","modified_gmt":"2023-10-05T17:50:23","slug":"mitre-engenuity-attck-evaluations-results","status":"publish","type":"post","link":"https:\/\/www2.paloaltonetworks.com\/blog\/2023\/09\/mitre-engenuity-attck-evaluations-results\/","title":{"rendered":"ONLY Cortex Delivers 100% Protection and Detection in MITRE Engenuity"},"content":{"rendered":"<h3>The 2023 MITRE Engenuity ATT&amp;CK Evaluation results are in, and only Cortex XDR delivers 100% protection and 100% analytic coverage with zero configuration changes and zero delayed detections.<\/h3>\n<p><a href=\"https:\/\/bcove.video\/45bcJEI\" rel=\"nofollow,noopener\" >Listen to the 2023 MITRE ATT&amp;CK Results<\/a><\/p>\n<p>The results of the fifth round of the MITRE Engenuity ATT&amp;CK Evaluations have been released, and while every vendor is claiming 100% of everything, the truth is in the details, and the data does not lie. This year, Cortex is the only vendor with 100% prevention and analytic detection.<\/p>\n<figure id=\"attachment_304530\" aria-describedby=\"caption-attachment-304530\" style=\"width: 915px\" class=\"wp-caption aligncenter\"><div style=\"max-width:100%\" data-width=\"915\"><span class=\"ar-custom\" style=\"padding-bottom:129.95%;\"><img loading=\"lazy\" decoding=\"async\"  class=\"wp-image-304530 lozad\"  data-src=\"https:\/\/www.paloaltonetworks.com\/blog\/wp-content\/uploads\/2023\/09\/word-image-304515-1.png\" alt=\"Results from MITRE Engenuity ATT&amp;CK Evaluation website, showing Palo Alto Networks, Microsoft, CrowdStrike and Sentinel One. \" width=\"915\" height=\"1189\" \/><\/span><\/div><figcaption id=\"caption-attachment-304530\" class=\"wp-caption-text\">An illustration directly from the <a href=\"https:\/\/attackevals.mitre-engenuity.org\/results\/enterprise?vendor=paloaltonetworks&amp;vendor=microsoft&amp;vendor=crowdstrike&amp;evaluation=turla&amp;scenario=1\" rel=\"nofollow,noopener\" >MITRE Engenuity ATT&amp;CK Evaluation<\/a> website shows the 2023 Turla evaluation results for the first detection test.<\/figcaption><\/figure>\n<p>In the ever-evolving landscape of cybersecurity, staying one step ahead has never been more critical. Adversaries continuously seek new and sophisticated ways to breach defenses. It's a high-stakes game where the rules constantly change, and the consequences of falling behind can be catastrophic.<\/p>\n<p>This is where MITRE Engenuity enters the scene with their Enterprise ATT&amp;CK Evaluations, offering a beacon of clarity amidst the chaos. These evaluations have become an invaluable resource for CISOs, security professionals and anyone tasked with safeguarding an organization's digital assets. They provide a litmus test or a performance review of how the industry\u2019s endpoint security solutions hold up in the face of the most resourceful cyber saboteurs.<\/p>\n<p>But, what exactly are the MITRE Engenuity ATT&amp;CK evaluations, and why should you care about the results? These annual evaluations emulate the tactics, techniques and procedures (TTPs) employed by some of our time's most active and notorious threat groups. The MITRE Engenuity red team invites cybersecurity solution providers to defend against carefully choreographed attacks (in both detect-only and prevent modes) to provide insight into three capabilities:<\/p>\n<ol>\n<li><strong>Visibility<\/strong> \u2013 what can a solution see<\/li>\n<li><strong>Detection<\/strong> \u2013 what actions can a solution accurately identify as malicious<\/li>\n<li><strong>Protection<\/strong> \u2013 what malicious actions can a solution prevent<\/li>\n<\/ol>\n<p>A staggering number of vendors participate in these evaluations \u2013 29 this year \u2013 which is a tribute to their value and reflects the painstaking efforts that MITRE Engenuity pours into the assessments to ensure they provide a challenging and provocative engagement.<\/p>\n<h2><a id=\"post-304515-_l9izqw8ruf95\"><\/a>Round 5 (Turla)<\/h2>\n<p>This year marks the fifth annual evaluation, and the MITRE Engenuity red team focused on emulating the methods of <a href=\"https:\/\/unit42.paloaltonetworks.com\/tag\/turla\/\">Turla<\/a>, a threat group our <a href=\"https:\/\/unit42.paloaltonetworks.com\/turla-pensive-ursa-threat-assessment\/\">Unit 42 threat researchers have studied<\/a>\u00a0extensively. Turla is an extraordinarily well-funded and sophisticated Russian-based threat group that has infected victims in over 45 countries. They have targeted government agencies, military groups, diplomatic missions, as well as research and media organizations. Turla\u2019s infamy stems from its covert exfiltration tactics, including water holing of government websites, custom rootkits, elaborate command-and-control network infrastructure and deception tactics. In speaking with defenders who participated, it is clear that MITRE Engenuity took a great leap forward in the sophistication of their attack methods this year.<\/p>\n<p>The blue team deployed the Cortex XDR Pro for Endpoint agent on both Windows and Linux endpoints. No additional solutions were deployed, and Cortex XDR was configured with default settings as it would be out of the box, with the only changes enabling the quarantining of malicious files and, for Linux, enabling the option to treat grayware as malware.<\/p>\n<h2><a id=\"post-304515-_35r1kvinaie9\"><\/a>Key Metrics and Evaluation Criteria<\/h2>\n<p>This year's evaluation was broken down into two detection-only scenarios, named Carbon and Snake, corresponding to notable tools created and used by Turla. A protection phase followed this, mirroring the techniques in the detection tests, with enough entropy injected not to look identical to the detection test. The two detection scenarios each had 10 steps, consisting of multiple substeps that map to actual techniques in the MITRE ATT&amp;CK framework. Combined, there were 143 substeps that each vendor had the opportunity to see. For each of these substeps, the MITRE Engenuity team recorded whether each solution had a detection for the action taken.<\/p>\n<p>Each of these detections was categorized based on the detection quality observed.<\/p>\n<p>MITRE Engenuity Detection Categories:<\/p>\n<ul>\n<li><strong>Not Applicable<\/strong> \u2013 Participants didn\u2019t have visibility on the system under test. (This would be the case if they opted out of the Linux evaluation.)<\/li>\n<li><strong>None <\/strong>\u2013 There was no detection. (No telemetry associated with the malicious activity)<\/li>\n<li><strong>Telemetry <\/strong>\u2013 Minimally processed data collected by the capability showing that event(s) occurred. (Detections of this type are usually basic logging of activity.)<\/li>\n<li><strong>General <\/strong>\u2013 The abnormal event was detected. However, an ATT&amp;CK tactic (or equivalent context) was not specified. (Detections of this type leave the security analyst to investigate and determine what action was taken and why.)<\/li>\n<li><strong>Tactic <\/strong>\u2013 An ATT&amp;CK tactic (or equivalent context) of the abnormal event was specified. (Detections of this type assert why an action occurred, but again leave the security analyst to investigate what action or technique was taken.)<\/li>\n<li><strong>Technique <\/strong>\u2013 An ATT&amp;CK technique (or equivalent context) of the abnormal event was specified. (Detections of this caliber provide the context and details required to answer why an adversary performed an action and precisely what action they used to achieve their objective.)<\/li>\n<\/ul>\n<p><div style=\"max-width:100%\" data-width=\"826\"><span class=\"ar-custom\" style=\"padding-bottom:57.14%;\"><img loading=\"lazy\" decoding=\"async\"  class=\"alignnone wp-image-304543 lozad\"  data-src=\"https:\/\/www.paloaltonetworks.com\/blog\/wp-content\/uploads\/2023\/09\/word-image-304515-2.png\" alt=\"Graph showing analytic coverage, minimally processed data and enriched detection. \" width=\"826\" height=\"472\" \/><\/span><\/div><\/p>\n<p>The MITRE Engenuity effectively identifies two types of coverage: Telemetry Coverage and \u201cAnalytic Coverage.\u201d Telemetry Coverage is defined as the number of substeps where a solution produces a Telemetry detection as its highest value detection. Analytic Coverage is defined as the number of substeps that contain either a General, Tactic or Technique detection.<\/p>\n<p>It is also worth noting that all detections might have one of two detection modifiers:<\/p>\n<ul>\n<li><strong>Configuration Changes<\/strong> \u2013 A configuration change modifier is noted if the detection observed was achieved on the fourth day of testing, a day when vendors are given a second chance to detect malicious activity they missed in the initial test.<\/li>\n<li><strong>Delayed Detections<\/strong> \u2013 A delayed detection modifier is noted if the detection is not observed in a timely manner, meaning a significant delay occurs when the action is taken, and when the detection is observed in the vendor\u2019s console.<\/li>\n<\/ul>\n<p>For the Protections scenario, there were 129 substeps organized into 13 major steps. In the case of these substeps. Each Protection substep was either blocked or not blocked and recorded as one of the following:<\/p>\n<ul>\n<li><strong>Protected<\/strong> \u2013 The malicious activity was blocked.<\/li>\n<li><strong>Not Applicable (Protected)<\/strong> \u2013 This occurred when a prior substep in the prevention step was blocked, and thus subsequent substeps could not be executed.<\/li>\n<li><strong>None<\/strong> \u2013 This means the malicious activity was not blocked.<\/li>\n<\/ul>\n<h2><a id=\"post-304515-_5hsjpddn2w4x\"><\/a>How Did Cortex XDR Perform?<\/h2>\n<p>The purpose of these evaluations is to provide insight into three capabilities:<\/p>\n<ol>\n<li><strong>Visibility<\/strong> \u2013 what can a solution see<\/li>\n<li><strong>Detection<\/strong> \u2013 what actions can a solution accurately identify as malicious<\/li>\n<li><strong>Protection<\/strong> \u2013 what malicious actions can a solution prevent<\/li>\n<\/ol>\n<p>Cortex XDR stands alone in providing 100% <strong>Protection <\/strong>while delivering 100% <strong>Visibility <\/strong>and 100% <strong>Analytic Coverage <\/strong>(detections) with zero configuration changes or delayed detections.<\/p>\n<p>In addition, the quality of our detections is unparalleled, with 142 of 143 detections as Technique level detections \u2013 <em>the highest-quality detection possible<\/em>. The one other detection was recognized as a Tactic level detection. Every one of the 129 substeps in the Protection evaluation was blocked. All of this was accomplished with zero \u201cconfiguration changes\u201d and zero \u201cdelayed detections.\u201d In fact, if we exclude detections resulting from a configuration change, Cortex XDR was the only vendor with no missed detections (detection type, None). In other words, Cortex XDR was the <em>only one with 100% visibility.<\/em><\/p>\n<figure id=\"attachment_304663\" aria-describedby=\"caption-attachment-304663\" style=\"width: 837px\" class=\"wp-caption aligncenter\"><div style=\"max-width:100%\" data-width=\"837\"><span class=\"ar-custom\" style=\"padding-bottom:65.95%;\"><img loading=\"lazy\" decoding=\"async\"  class=\"wp-image-304663 lozad\"  data-src=\"https:\/\/www.paloaltonetworks.com\/blog\/wp-content\/uploads\/2023\/09\/image-15.png\" alt=\"Cortex screenshot of the MITRE Engenuity ATT&amp;CK Evaluations Dashboard.\" width=\"837\" height=\"552\" \/><\/span><\/div><figcaption id=\"caption-attachment-304663\" class=\"wp-caption-text\">The <a href=\"https:\/\/app.powerbi.com\/view?r=eyJrIjoiNWRhYzY1YjItOTAxZC00MGM5LThlNzYtOTYxNzViYzM1ZGY2IiwidCI6IjgyOTNjZmRmLThjMjQtNDY1NS1hMzA3LWVhMjFjZDNiMjJmZiIsImMiOjF9\" rel=\"nofollow,noopener\" >Palo Alto Networks MITRE Engenuity ATT&amp;CK Evaluations Dashboard<\/a>. Snapshot showing results from the 2023 Turla evaluation.<\/figcaption><\/figure>\n<p>The results from this year\u2019s evaluation serve to reflect the tremendous amount of effort that Palo Alto Networks continues to pour into both adversarial research and engineering in endpoint security, putting that knowledge to work to help our customers remain safe in the face of an increasingly hostile cyber world.<\/p>\n<h2><a id=\"post-304515-_cyf6ku2c9vmj\"><\/a>A Note about 100%<\/h2>\n<p>If you\u2019ve been following the ATT&amp;CK evaluations for some time, 100% might seem commonplace to you. But don't get confused; when other solutions claim to have achieved 100% in this year's evaluation, they have at least one detection and\/or prevention in all the major steps. Only Cortex provided these detections for every one of the individual malicious actions that were taken \u2013 the substeps.<\/p>\n<p><span style=\"font-weight: 400;\">100% detection and prevention is the bar we should all strive for. As Forrester\u2019s Allie Mellen pointed out after last year\u2019s evaluation, \u201c<\/span><a href=\"https:\/\/www.forrester.com\/blogs\/mitre-attck-evals-getting-100-coverage-is-not-as-great-as-your-vendor-says-it-is\/\" rel=\"nofollow,noopener\" ><span style=\"font-weight: 400;\">detecting an attack inspired by a known threat actor should be the floor for what security products can do, not the ceiling.<\/span><\/a><span style=\"font-weight: 400;\">\u201d<\/span><span style=\"font-weight: 400;\">1<\/span><span style=\"font-weight: 400;\"> We completely agree, and Cortex XDR is setting the bar.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">That said, Allie also rightly pointed out that perhaps 100% detection isn\u2019t necessarily a good thing as solutions that \u201c<\/span><a href=\"https:\/\/www.forrester.com\/blogs\/mitre-attck-evals-getting-100-coverage-is-not-as-great-as-your-vendor-says-it-is\/\" rel=\"nofollow,noopener\" ><span style=\"font-weight: 400;\">are detecting on every technique, they have the potential to be noisy, have a high rate of false positives, and provide excessive alerting.<\/span><\/a><span style=\"font-weight: 400;\">\u201d We also share this perspective, and this is why we heavily leverage machine learning to understand which activities represent normal operations and which are potentially malicious actions. In the case of Cortex XDR, this is only strengthened by the fact that we integrate additional telemetry from many other key data sources that critical context clues to help determine where you should focus your investigations.\u00a0<\/span><\/p>\n<h2><a id=\"post-304515-_g8wcqh4uztvk\"><\/a>Exploring the ATT&amp;CK Evaluation Results<\/h2>\n<p>Hats off to the MITRE Engenuity, as this year was significantly more challenging than years past. In addition, we are thrilled with the improvements they have made to help defenders make informed decisions. The ATT&amp;CK evaluation results, hosted on MITRE Engenuity\u2019s site, now allow you to compare the <a href=\"https:\/\/attackevals.mitre-engenuity.org\/results\/enterprise?evaluation=turla&amp;scenario=1\" rel=\"nofollow,noopener\" >results for three vendors side by side<\/a>. You can choose each of the three evaluation scenarios and up to three vendors. By default, the results are shown with delayed detections and those resulting from configuration changes included. Still, you can also view the results with detections with those modifiers removed.<\/p>\n<figure id=\"attachment_304569\" aria-describedby=\"caption-attachment-304569\" style=\"width: 1424px\" class=\"wp-caption alignnone\"><div style=\"max-width:100%\" data-width=\"1424\"><span class=\"ar-custom\" style=\"padding-bottom:78.65%;\"><img loading=\"lazy\" decoding=\"async\"  class=\"wp-image-304569 lozad\"  data-src=\"https:\/\/www.paloaltonetworks.com\/blog\/wp-content\/uploads\/2023\/09\/word-image-304515-4.png\" alt=\"MITRE Enenuity ATT&amp;CK Evaluation results, showing Palo Alto Networks, Microsoft, CrowdStrike and Sentinel One.\" width=\"1424\" height=\"1120\" \/><\/span><\/div><figcaption id=\"caption-attachment-304569\" class=\"wp-caption-text\">From the <a href=\"https:\/\/attackevals.mitre-engenuity.org\/results\/enterprise?vendor=paloaltonetworks&amp;vendor=microsoft&amp;vendor=crowdstrike&amp;evaluation=turla&amp;scenario=1\" rel=\"nofollow,noopener\" >MITRE Engenuity ATT&amp;CK Evaluation website<\/a>, the protection evaluation results. Only Cortex XDR had 100% prevention.<\/figcaption><\/figure>\n<p>We\u2019ve heard for many years that it is difficult to understand the ATT&amp;CK evaluation results as no data visualization tool allows you to view the results across all vendors. To aid in that effort, we\u2019ve created a tool to help you explore this year\u2019s results, as well as all previous years. With this tool, you can select the vendors you are interested in and the particular adversaries that have been emulated. All of the resulting charts are built directly from the JSON files provided by MITRE Engenuity for each participating vendor. Explore the <a href=\"https:\/\/app.fabric.microsoft.com\/view?r=eyJrIjoiNDk5ZWFmODctYjY2ZS00NzI5LWJkZDYtNDE5ODAwYjU2ZGNlIiwidCI6IjgyOTNjZmRmLThjMjQtNDY1NS1hMzA3LWVhMjFjZDNiMjJmZiIsImMiOjF9\" rel=\"nofollow,noopener\" >ATT&amp;CK Evaluation results<\/a> for yourself and let the data reveal the best choice for defending your business.<\/p>\n<figure id=\"attachment_304582\" aria-describedby=\"caption-attachment-304582\" style=\"width: 1241px\" class=\"wp-caption alignnone\"><div style=\"max-width:100%\" data-width=\"1241\"><span class=\"ar-custom\" style=\"padding-bottom:35.46%;\"><img loading=\"lazy\" decoding=\"async\"  class=\"wp-image-304582 lozad\"  data-src=\"https:\/\/www.paloaltonetworks.com\/blog\/wp-content\/uploads\/2023\/09\/word-image-304515-5.png\" alt=\"Graph showing technique level detections versus percentage of substeps blocked. \" width=\"1241\" height=\"440\" \/><\/span><\/div><figcaption id=\"caption-attachment-304582\" class=\"wp-caption-text\">The <a href=\"https:\/\/app.powerbi.com\/view?r=eyJrIjoiNWRhYzY1YjItOTAxZC00MGM5LThlNzYtOTYxNzViYzM1ZGY2IiwidCI6IjgyOTNjZmRmLThjMjQtNDY1NS1hMzA3LWVhMjFjZDNiMjJmZiIsImMiOjF9\" rel=\"nofollow,noopener\" >Palo Alto Networks MITRE Engenuity ATT&amp;CK Evaluations Dashboard<\/a>. A snapshot showing results from all 5 years of evaluations, plotting both Technique Level Detections and Substeps Blocked.<\/figcaption><\/figure>\n<ol>\n<li><em><span style=\"font-weight: 400;\">Forrester\u00ae, and Allie Mellen. \u201cPrincipal Analyst.\u201d MITRE ATT&amp;CK Evals: Getting 100% Coverage Is Not As Great As Your Vendor Says It Is, Forrester\u00ae, 2022, https:\/\/www.forrester.com\/blogs\/mitre-attck-evals-getting-100-coverage-is-not-as-great-as-your-vendor-says-it-is\/. Accessed 19 September 2023.<\/span><\/em><\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>MITRE Engenuity ATT&#038;CK Evaluations have been released. This year, Cortex is the only vendor with 100% prevention and analytic detection.<\/p>\n","protected":false},"author":645,"featured_media":304706,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[308,6717],"tags":[6737,8925],"coauthors":[6788],"class_list":["post-304515","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-announcement","category-products-and-services","tag-cortex-xdr","tag-mitre-engenuity","sec_ops_category-must-read-articles","sec_ops_category-news-and-events"],"jetpack_featured_media_url":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-content\/uploads\/2023\/09\/MITRE_Attack-Eval_TURLA_Badge-4.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/posts\/304515","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/users\/645"}],"replies":[{"embeddable":true,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/comments?post=304515"}],"version-history":[{"count":11,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/posts\/304515\/revisions"}],"predecessor-version":[{"id":304595,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/posts\/304515\/revisions\/304595"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/media\/304706"}],"wp:attachment":[{"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/media?parent=304515"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/categories?post=304515"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/tags?post=304515"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www2.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/coauthors?post=304515"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}