Risk Assessment Report: A Focus on Office 365

What Office 365 misses may surprise you.

We now have been running Email Security Risk Assessment (ESRA) tests, collecting and analysing the data and publishing reports for over two years!

In December 2018, I focused on the overall false negative rate that we had seen across all of the incumbent email security systems we had tested against to that point. It was quite a diverse list of incumbent systems.

This time, I am going to focus on the data we have collected over these two years specifically with Microsoft Office 365™ (Exchange Online Protection or Advanced Threat Protection) as the incumbent email security system. But before I do that, for those readers new to ESRA testing, I first need to explain what it is all about.

How Does the ESRA Work?

In an ESRA test the Mimecast Secure Email Gateway service reinspects a participating organisation’s emails that were deemed to be safe by their incumbent email security system. This is based on actual inbound email traffic, not on test emails. We run this test over a period of time, usually between a week and a month at each organisation. An ESRA test passively inspects and records the security results of real emails that have been delivered to their employees.

In security terms, an ESRA test is a false negative hunting test, where the Mimecast email security service inspects delivered emails for missed spam, phishing, malicious files and URLs, and impersonation emails.

Before I get into the Office 365-specific results, it’s worth noting that we recently added the detection of malicious URLs within delivered email to our ESRA testing capability. In aggregate, our ESRA testing has detected 463,546 malicious URLs that were contained in 28,407,664 delivered emails. This comes out to an average of one malicious URL getting through an organisation’s email defenses for every 61 delivered emails. Given how many emails a typical organisation gets in a day, that is a lot of malicious URLs waiting to be clicked in employees’ inboxes!

Office 365 Misses a Variety of ‘Bad’ Emails

Now to the Office 365-specific results:

• Of the 232 million emails we have inspected in aggregate, 105 million — or almost half — of those had passed through Office 365 as the incumbent email security system.
• Of the 75 organisations for whom we have conducted ESRA testing, more than half, or 47 of them used Office 365 as their incumbent security system.
• We have found that Office 365’s false negative rate for spam to be 16% (as in 16% of delivered email was actually false negative type spam) versus 11% for ESRA testing across all incumbent security systems. That may not seem like a big difference, but to an organisation receiving a lot of spam, it can be quite burdensome.
• We also found that Office 365 let in more than its fair share of impersonation attacks, more than 33,000 of them, as well as unwanted, potentially dangerous or malicious file attachments — also more than 33,000 of them.

We promise to keep testing and to keep reporting on what we find. While perfect security is not possible, better security most certainly is. And we see our ESRA testing as a great way to keep the focus on false negatives and how best to minimise them. So, stay tuned for our regular, quarterly reports and the insights they offer.