When the term “big data” is bandied about these days, few consider the actual volume of data being created. In a recent Forbes article, IDC predicts the world will be creating 163 zettabytes of data per year by 2025. This is a ten-fold increase in worldwide data creation from the current rate of 16.3ZB.
With all this data being collected from multiple applications and infrastructures, storage and processing presents a real challenge for IT professionals – not to mention a burdensome cost. The options for dealing with this large volume of data are less than optimal. Exporting data from the cloud to a physical datacenter is exorbitant. Cloud storage of log data requires a significant amount of space, which has prompted many administrators to cull logs in order to reduce the volume and subsequent cost of storage. Of course, winnowing this log data can lead to the loss of valuable and irreplaceable information. And ultimately, the loss of data can diminish the potential benefits and advantages of big data.
As more and more log data is collected from sources such as load balancers, other network equipment, servers, databases, and service enablers, IT and commercial decision-makers find themselves unable to access this vital information in real time. And without real-time access, they cannot assure the performance of critical services and applications.
IT needs to move beyond log data and take a smart data approach. Such an approach zooms in on the wire data extracted at the source (such as IP packets, segments, sessions, and application data flows) where it is compressed into metadata in real time. Because compression levels are high, and only relevant information is retained, storage costs are significantly reduced. This also enables IT to keep this data longer, which allows for detailed historical analysis of incidents and events.
By continuously monitoring wire data across key service performance metrics, IT gains access to completely contextualized data in real time for the entire IT infrastructure. Using a smart data approach, companies can benefit from greater visibility into their networks, allowing IT to assure the quality of service and business operations. In addition to reducing the size of storage, smart data allows companies to have access to the valuable information they need to leverage meaningful, actionable insights that support better decisions and achieve business success.
This blog is based on the article, From log data to smart data written by Daniel Crowe, Regional Director France and Southern Europe for NETSCOUT, which was published at Solutions Numeriques.