MSPs and Big Data: Why and How

By Vikas Aggarwal 2

Always regarded as a non-critical part of day-to-day operations in the past, data and its delayed analysis was relegated to batch processing tools and monthly meetings. Today, as the IT industry has snowballed into a fast moving avalanche of Cloud, virtualization, out-sourcing and distributed computing, the science of extracting meaningful intelligent metrics from Big Data has become an important and real-time component of IT Operations.

No longer do IT management systems work in vertical or horizontal isolation as just a few years ago. The inter-dependence between IT Services, applications, servers, cloud services and network infrastructure has a direct and measurable impact on Business Services.

Why Big Data in Cloud Performance Tools

The amount of data generated by these components is huge and the rate at which this data is generated is so fast that traditional tools cannot keep up with any kind of real time correlation. The combined volume of data generated by this hybrid infrastructure can be huge, but if it is correlated properly, it can give misson critical insight into:

This data has to be analyzed and processed in real-time in order to provide proactive responses and alerting for service degradation. The data that is being collected can be structured or unstructured, coming from a variety of systems which depend on each other to offer optimal performance, and has little to no obvious linkage or keys to one another (i.e. the data coming from an application is completely independent of the data coming from the network that it is running on). Some examples of data sources that need to be correlated are application logs, netflow, JMX, XML, SNMP, WMI, security logs, packet analysis, business service response times, weather, news, etc.

Managed Service Providers are moving to hybrid cloud environments themselves and offering services ranging from security, backup, VoIP, applications and compute resources. Also, enterprises are outsourcing more and more of their IT peformance management, and they expect the MSP to handle any and all kinds of IT data. Managed Service Providers must adopt monitoring systems that are flexible and can handle Big Data efficiently. Such IT monitoring platforms will allow them to offer higher value added services to enterprise customers. Once they have such versatile Big Data systems in place, they can offer real-time responses to alarms and alerts and give meaningful business impact analysis to these enterprise customers.

Contextual analytics and presentation of data from multiple sources is invaluable to IT Operations in troubleshooting poor application performance and user satisfaction. As a simple example, a user response time application could send an alert that the response time of an application is too high. Application Performance Monitoring (APM) data could indicate that a database is responding slowly to queries because the buffers are starved and the number of transactions is abnormally high. Integrating with network netflow or packet data would allow immediate drill down to isolate which client IP address is the source of the high number of queries.

How to Handle Big Data for Cloud Performance

Traditional monitoring or BI platforms are not designed to handle the volume and variety of data from this hybrid IT infrastructure. The management platforms need to be designed to correlate Big Data from the IT components in real-time and provide feedback to the operations team for proactive responses. As these monitoring systems evolve, their Big Data correlation components will become richer and more analytical and will position these MSPs for the IT environments of the future.

New generation MSP monitoring solutions that are scalable, have predictive analytics, multi-tenant and a granular security model are now available from a small number of vendors. Single use systems that are designed for just network data or just application data are trapped within the same boundaries that makes Big Data meaningless – by its very nature, Big Data systems need to be able to handle a very wide variety of data sources to provide greater uptime from faster troubleshooting and lower OpEx from correlated analysis.

Vikas Aggarwal is VP of Product Marketing at Kaseya.