Big data is getting bigger, and so are its security problems

Big data, by virtue of its size and potential, is both a weapon and a target in cybersecurity terms.

Its insights can give organisations the edge on their competitors, yet it also contains information that criminals can leverage in fraud and ransomware scams.

Thankfully, cybersecurity teams can use big data to optimise their operations in much the same way that its insights can help other parts of the business. Used correctly, big data can aid threat analysis and response, help your organisation prioritise vulnerabilities, and let you get inside your attackers’ heads.

The problem? Well, it’s in the name: big data’s volume can overwhelm analysts, especially if it arrives in different forms across various platforms. The solution? AI, machine learning and intelligent software can help security teams manage and profit from big data.

What is ‘big data’?

The Australian Cyber Security Centre (ACSC) defines Big Data as large amounts of structured and unstructured data that exceeds the ability of commonly used software tools to capture, manage and process. Big data requires combining techniques and technologies with new forms of integration to reveal insights from datasets that are diverse, complex, and of a massive scale.

Massive might just be an understatement. The World Economic Forum estimates by 2025 we’ll be creating 463 exabytes of data daily. By comparison, you could store all the words ever spoken by humans in five exabytes.

Big data gives cybercriminals a big opportunity

Individual organisations may not reach exabyte scales, but they still churn a lot of data. A medium-sized network with about 20,000 devices (including laptops, smartphones and servers) transmits over 50 terabytes of data in a day – an average of five gigabits per second.

In addition to its sheer volume, this data moves through ever-more complicated digital webs. The traditional end points of cybersecurity have been joined by cloud-hosted resources and Internet of Things (IoT) devices. These connections lead to a dramatically increased attack surface. And cyber criminals are hard at work trying to exploit them.

Hackers are using machine-learning technology to automate information gathering and identify security weaknesses, helping their gangs launch devastating attacks. Old-fashioned threat detection tools, intrusion response and firewalls simply can’t keep pace with the adaptable nature of modern cyberattacks.

Big data also contains the security solution

As we’ve seen, even a medium-sized network can generate a lot of data. The data itself is simply an asset: they key lies in how it’s used. Use it as a security tool, and you’ve got power in your hands:

1. analysing current and historical data can help you predict future attacks, identify common modes of attack and better understand attacker behaviour

2. data can help you build insights to sift genuine threats from false alarms

3. by comparing and linking anomalies, you can automate an effective response to attacks – a quicker and more cost-effective approach than manual measures

Big-data analytics can be a gamechanger. But digesting this data, making it visible and making it available to the right person at the right time is a gargantuan task. It’s no surprise that most companies only leverage a small proportion: according to one survey, two-thirds of data goes unused. The same applies to security. It takes skilled humans equipped with the appropriate tools to glean useful cybersecurity insights from the mass of available data.

In a security context, the volume of data that comes from monitoring software, logs, ticketing and case management systems can be overwhelming. The alert fatigue often felt by analysts may be exacerbated by a steady stream of threat news from government, industry and partner sources. And as networks grow in size and complexity, relying on manual methods to manage alerts is like trying to drink water from an overpowered firehose.

Left unmanaged, the problem will spiral. As new threats emerge, new security solutions and new threat feeds need to be put in place – leaving cyber professionals so wrapped up in sorting out the noise that they can barely wriggle free to do their jobs. Meanwhile, if threat analytics are not effectively contextualized and refined, threat data will return numerous false positives, bogging your cybersecurity team down even further.

Automation and integration can help upgrade your security

Given these pressures, building a robust strategy based on real-time, data-rich insights will be a priority for many organisations. Part of the answer is automation:

1. Intrusion Detection Systems (IDS) monitor suspicious activity and may be able to mount an automated response

2. Security Information and Event Management (SIEM) software and Endpoint Protection Platforms (EPP) provide continuous monitoring across infrastructure and endpoints

3. Security Orchestration, Automation and Response (SOAR) platforms can help streamline responses and integrate systems

Insights are worthless unless they are both timely and actionable. A single incident, whether suspicious user activity or usual network traffic, doesn’t provide enough information to mount a proper defence. To build an accurate picture of threats and vulnerabilities, you must be able to consolidate these insights into usable information. That’s why advanced SIEM systems use machine learning to turn unstructured data from disparate sources into user-friendly analytics. Extended Detection and Response (XDR) is an increasingly popular silo-busting approach that seeks to offer across-the-board data visibility. This consolidated data can help remove false positives, identify attack patterns and automate many security monitoring tasks.

How Big data can inform your cybersecurity strategy

When configured correctly, big data analytics will be able to monitor events across different sources and combine them to provide a holistic threat view to your team. In an ideal case, big data analytics will:

1. normalise data monitoring and provide context if unusual activity is detected

2. remove redundant information and prioritise what’s shown based on your business goals

3. present the data in a format that works for the teams that require it

Just as monitoring and analysis should be amalgamated to build a bigger cybersecurity picture, big-data analytics must be complemented by wider cybersecurity best practices, such as using credentials to manage user access and running frequent awareness training to limit the dangers from social engineering attacks. The key is to align Big Data analytics with your cybersecurity goals. By having a clear picture of the kind of insights you need, you can ensure your team gets the most relevant information.

Set it up the right way, and big data can protect itself (mostly)

With enough time and large enough data sets, machine learning and AI will kick into high gear and be able to forecast, identify and resolve a large number of threats all on their own. The insights from historical data and current data will help you spot any dangerous gaps in your security and create a plan to strengthen them.

While it does require an upfront investment in time and resources to set up and configure that level of automation, the rewards are well worth it, especially for organisations of a certain size and complexity. Using big data-analytics to power your security efforts will allow your organisation to move beyond traditional security tools, respond more quickly to threats, and offload some of the burdens on already-stretched security teams.