MITRE ATT&CK Evaluations 2023: Palo Alto, Microsoft, CrowdStrike & Cybereason Lead the Way
We Keep you Connected
MITRE ATT&CK Evaluations 2023: Palo Alto, Microsoft, CrowdStrike & Cybereason Lead the Way
eSecurity Planet content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
MITRE Engenuity has released its 2023 ATT&CK evaluations, examining how top cybersecurity vendors detect and prevent sophisticated cyberthreats. This year, the evaluations focused on the techniques of Turla, a Russia-based threat group.
Turla uses a command-and-control network, as well as open source tools, which are more difficult to protect and easier to exploit because anyone can edit — and abuse — the code.
This year’s MITRE analysis tested vendors’ ability to detect two scenarios called SNAKE and CARBON. MITRE used multiple offensive security tools, including Keylogger and Mimikatz, to launch attacks on vendors’ environments. The vendors were also tested on protection capabilities, undergoing thirteen tests — some with many steps — to see at which step they could halt an attack.
MITRE’s detection and protection evaluations have usually attracted endpoint security vendors, with the detection evaluations best suited for endpoint detection and response (EDR) products and the protection tests focusing on the abilities of endpoint protection platforms (EPP). Vendors offering both EDR and EPP capabilities for Windows and Linux are able to participate in more steps of an evaluation than vendors with more limited offerings. Over time, security vendors whose primary strengths lie elsewhere have increasingly participated in the respected program.
We encourage security buyers to research these vendors, including their MITRE scores over time, before making a purchase. Our analysis provides one way to look at the MITRE evaluations from an angle that can be helpful. But as we noted in our analysis of last year’s results, your organization will need to test security products in your own infrastructure before you know if it will work for you. And looking into the details of the MITRE tests may also give you significant information about how a product might perform in your environment.
The MITRE results were separated into two categories: detection (SNAKE and CARBON scenarios) and protection (13 tests of a product’s ability to stop an attack).
The detection evaluation involved 143 total steps. For vendors who skipped the Linux tests, that number drops to 132. To calculate the detection scores, we divided the total number of successfully detected steps by 143 (or 132).
The detection tests include an analytics score, a telemetry score, and a visibility score. Analytics coverage not only detects the threat but also tags it with the MITRE standard identification. Telemetry coverage is the collection of raw data about a threat event, not necessarily including context. Visibility coverage is the overall number of detections that MITRE tested and the vendor successfully detected. We cover only the visibility score in our analysis of MITRE testing.
Cisco’s and Check Point’s detection and protection scores weren’t recorded due to technological issues, according to MITRE.
The protection component consists of 13 tests, evaluating which vendors can stop all thirteen attack sequences and how quickly they can do so. Most of the 13 tests had multiple attack steps. Protection tests were optional, and not all vendors participated, including Rapid7 and WithSecure.
On the protection side, there were a few tests in particular with which multiple vendors struggled. Many vendors missed test three, including Fortinet, Bitdefender, and Sophos. Malwarebytes couldn’t complete it either.
And many struggled with test seven as well. While Fortinet eventually completed seven, it took many steps. Same for Bitdefender. VMware Carbon Black didn’t complete test seven, and Tehtris missed every step. A few vendors ran into trouble with test 13 as well. This year’s evaluations revealed some common threads, with a few notable protection tests that appeared to be particularly hard. Palo Alto Networks had a perfect score, detecting all 143 detection tests and stopping all 13 protection evaluations on the first step. Three other vendors — Microsoft, CrowdStrike and Cybereason — all successfully detected the 143 detection tests and stopped the 13 protection attacks, but missed a small number of the protection steps before stopping the threat.
A number of other vendors had strong showings, but the results for many vendors left room for improvement, especially on the protection end. Many high-profile security vendors failed to stop multiple protection tests. Product teams typically use the tests to improve their offerings, so participation is always a net positive.
Vendors have noted a number of caveats about the evaluations — namely, that the detection tests can potentially be gamed by vendors setting detection sensitivity levels high enough to produce false alerts in the real world, and some vendors have said they had to disable key securityfeatures to participate. Those caveats make the protection tests the more important of the two, and many welcomed them when they were introduced two years ago.
Many of last year’s winners scored high for detection again this year, including Palo Alto, Microsoft, CrowdStrike, and Cybereason. Five vendors received perfect visibility scores in the detection evaluations, and Sophos was one vendor that scored a noteworthy comeback from a middling 2022 result, underscoring that vendors often use the results to better their products.
The following table gives the overall visibility score for each vendor in the detection tests, from highest number of successful detection tests to the lowest.
The protection test results this year displayed a wide performance range. Only seven vendors stopped all the tests they faced, the same number as last year. But many vendors missed multiple protection tests entirely. There were 13 tests total, and many contained multiple steps. This table compares three different items:
These metrics show different facets of the protection tests, rather than a single overall percentage.
Palo Alto stopped all the tests, once again earning our confidence as the top overall cybersecurity vendor. Symantec and Cybereason did particularly well here. Malwarebytes and Tehtris were the lowest performers on the protection side, only managing to stop 7 of 13 tests. And while Fortinet stopped 12 of the tests, it missed a sizable 43 attack steps during the testing process. The deeper intruders get into your environment, the more damage they can do.
MITRE evaluations are far from easy for security vendors, and that difficulty makes them particularly valuable in a market where buyers don’t have a lot of visibility. Even vendors who choose to undergo only the detection tests, like Rapid7, are to be commended for pursuing excellence in a field — like EDR — that they aren’t best known for.
We encourage you to study the MITRE results for yourself if you’re interested in knowing more or are considering making a purchase from one of these vendors. Our interpretation is just one method of looking at the data.
Threats only grow more sophisticated over time, and security providers have the difficult task of keeping up with threat actors’ ingenuity. That makes MITRE evaluations one of the best available tools for both security buyers and vendors to learn.
Strengthen your organization’s IT security defenses by keeping up to date on the latest cybersecurity news, solutions, and best practices.
Strengthen your organization’s IT security defenses by keeping abreast of the latest cybersecurity news, solutions, and best practices.
Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.
Enhanced Expertise: Co-Managed services bring in specialized expertise to complement your IT team, helping them tackle complex issues and projects more effectively.
Resource Augmentation: It's not about replacing your IT department but augmenting their resources. This allows your IT team to focus on strategic initiatives while routine tasks are handled externally.
Scalability: Co-Managed services are scalable, so you can adjust the level of support as per your needs, ensuring efficient resource allocation.
Cybersecurity Boost: Co-Managed services often provide advanced cybersecurity solutions, which help protect your organization from cyber threats and vulnerabilities.
Cost-Efficiency: By outsourcing routine tasks and maintenance, your IT department can allocate resources more efficiently, potentially reducing overall IT costs.
Improved Compliance: Co-Managed services can assist with compliance management, ensuring your organization adheres to industry regulations and standards.
Risk Mitigation: Shared responsibility for IT operations means shared risk. Co-Managed services providers work alongside your IT team to minimize potential risks.
Strategic Partnerships: Partnering with experienced Co-Managed service providers can enhance your organization's reputation by showcasing a commitment to innovation and efficiency.
Faster Issue Resolution: Co-Managed services often have access to advanced tools and resources, enabling quicker problem-solving and issue resolution.
Customized Solutions: Tailored solutions mean that your IT department has more control over the services provided and can align them with your organization's specific needs.
Flexibility: Your IT team retains control and can collaborate closely with Co-Managed service providers, ensuring a seamless partnership.
Catering to All IT Issues So You Can Stay Connected Securely
The Network Company has been based in South Orange County, CA, for over 27 years and provides “Managed IT Services.” We support your company’s network, computers, software, and users; and make sure your system is always running smoothly. Our topmost priority is to ensure that your users and customers get the most from your IT investment.
GET YOUR FREE, NO-OBLIGATION NETWORK HEALTH CHECK! We know you’re so busy running your business that sometimes you may forget to think about the security and health of your computer network. In fact, many business owners do NOT perform regular IT and Security maintenance, leaving the door wide open for spyware, viruses and other malicious threats that can infect their networks. This can lead to the loss of irreplaceable business data and hours of downtime. This is where we can help with Professional IT services, no matter what industry your business is in.
We don’t want this to happen to you! We’re offering you a FREE, no-strings-attached Network Health Check, which includes an inventory of your current environment, along with recommended improvements to keep your network healthy.
What’s the catch? You must be wondering why we are willing to give this away for free. We are simply offering this Network Health Check as a risk-free way to “get to know us” while helping you identify areas of vulnerability.
How does it work? To get your free Network Health Check, simply click here to complete the online request form. After we receive your request, we will contact you to schedule a specialist to perform the assessment.
Following the assessment, you will receive a complimentary recommended action plan and estimate for correcting any existing issues.