OpsClarity Promises Easier Big Data Management for DevOps

What will it take to make open source big data tools truly useful for the enterprise? OpsClarity thinks the answer is a one-stop solution for monitoring everything from Spark to Elasticsearch to MongoDB. That’s what it rolled out this week in a new platform targeted at DevOps teams.

The platform, called Intelligent Monitoring, is designed to help DevOps teams understand how their various big data processing tools are interacting with each other, as well as track performance. It supports all of the major big data applications, including Apache Kafka, Apache Storm, Apache Spark as well as datastores such as Elasticsearch, Cassandra and MongoDB.

OpsClarity is pitching the product as a solution to the challenge created by the rapidly expanding feature sets and sophistication of open source big data tools.

“Technologies like Apache Kafka, Apache Spark and Apache Storm are constantly expanding the scope of what is possible,” the company said in a statement. “However, most of these data processing frameworks are themselves a complex collection of several distributed and dynamic components such as producers/consumers and masters/slaves. Monitoring and managing these data processing frameworks and how they are dependent on each other is a non-trivial undertaking and usually requires an extremely experienced operations expert to manually identify the individual metrics, chart and plot them, and then correlate events across them.”

OpsClarity hopes companies will choose to invest in its product rather than rely on employees to try to keep track of big data operations.

The company has a point when it says open source big data is growing too complex, and too quickly, to be managed effectively the old-fashioned way. Just a few years ago, NoSQL was still a new phenomenon, and the only open source big data platforms that most people had heard of were Hadoop, Spark and Elasticsearch. Now that list is much longer, and it continues to expand as developers introduce new ways of leveraging hardware features, as well as software-defined infrastructure, for big data processing.

And that’s an important message for VARs. Open source big data tools are great, but they’re also very complex. If OpsClarity is right, customers will pay for big data solutions that help them track what’s going on, without really having to understand how everything works under the hood.