Consider modern enterprise IT infrastructure. Increasingly, it is a complex combination of on premise computing and storage and off premise, cloud-based resources. Tying all of this together is a web of data connections. Applications can run either in the cloud or locally, and all of this is subject to penetration by bad actors. Combine this with the internet of things (IoT), where virtually every device is connected, and the number of points of potential compromise increase exponentially. The wonder is not that so many large enterprise networks are breached, but that so few are.

It is no wonder that the security industry is burgeoning. With five-year growth estimated by Forbes at 9.8 percent annually, security has become big business. As the threat becomes more acute, the security solutions being introduced to the market are also becoming more sophisticated. The current notion is that single solutions are no longer sufficient to adequately protect the enterprise: “defense in depth” is the new mantra—where multiple solutions beginning at the edge of the network and becoming increasingly more complex the closer to the application one gets—becomes the norm.

There is a point of diminishing returns to all this, of course: Each security solution comes with additional complexity and, usually, additional personnel to manage the new application. Ultimately, security consumes a disproportionate share of IT resources and ultimately an organization begins to treat security as a statistical game: where additional security is weighed against the probability of a breach. When the bad guys out number the good guys, this is a losing game.

Yet, there may be a light at the end of this tunnel and it may be a result of technology that, itself, may pose the greatest threat to security: big data.

, using cluster technology such as Hadoop Distributed File System (HDFS), application logs can be collected and analyzed to determine if an attack is taking place. Experiments confirm that such an approach is at least as effective as more pervasive approaches.

Experiments such as those described by Win et al. provide hope that, ultimately, big data will be the solution that enables cost efficient security services that can keep up with both network complexity and advances in the threat.

It’s a big data world, after all

Ultimately, big data will be an essential component to every IT environment. Along with the cloud and advanced analytics, big data forms the backbone of the evolving computing landscape. Enabling this environment is data networking and, as a result, security.

Just as in previous IT environments, where the infrastructure, itself, provided the security solutions that sought to prevent breaches, the new architectures will depend on big data and advanced analytics to detect and mitigate threats. Once again, big data is increasingly enabling new approaches to computing: ones characterized by the use of large data sets in a near real time environment.

This article is published as part of the IDG Contributor Network.