Cleversafe Plans to Combine Storage and Massive Computation

Cleversafe Inc. announced plans to build a Dispersed Compute Storage solution by combining Hadoop MapReduce with Cleversafe’s scalable Object-based Dispersed Storage System.

This solution will alter the big data landscape by decreasing infrastructure costs for separate servers dedicated to analytical processes, reducing required storage capacity, and simultaneously improving data integrity.

In addition, the company’s solution will reduce network bottlenecks by bringing together computation and storage at any scale, petabytes to exabytes and beyond.

Traditional storage systems are not designed for large-scale distributed computation and data analysis. Present implementations treat data storage and analysis of that data separately, transferring data from SANs or NASs across the network to perform the computations used to gather insight. In this manner the network quickly becomes the bottleneck, making multi-site computation over the WAN particularly challenging.

Cleversafe solves this problem by combining Hadoop MapReduce alongside its Dispersed Storage Network (dsNet) system on the same platform and replacing the Hadoop Distributed File System (HDFS) which relies on three copies to protect data with Information Dispersal Algorithms thereby improving reliability and allowing analytics at a scale previously unattainable through traditional HDFS configurations.

"For any company, the movement, management and storage of massive data stores for analytical purposes is already unmanageable," said Chris Gladwin, CEO and president of Cleversafe. "Many companies have had to invest significant resources in both CAPEX and OPEX to manage the challenge of big data and to try and capitalize on the opportunity to gather insights from that data."

"The key to reducing both cost and complexity is to combine computation with dispersed storage," said Gladwin. "Cleversafe’s solution will provide infinitely scalable, reliable, and cost effective storage for data to support massive computation while enhancing the analysis workflow."

Hadoop MapReduce, which is being used broadly throughout the industry, represents only a partial solution to this problem. While it lends itself naturally to enabling computations where the data exists rather than transferring data to computation nodes, it has inherent scalability and reliability limitations. Current HDFS deployments utilize a single server for all metadata operations and three copies of the data for protection. Failure of the single metadata node could render stored data inaccessible or result in a permanent loss of data. Maintaining three copies of data at massive scale for protection leads to skyrocketing overhead and management costs.

Cleversafe’s dsNet system protects both data and metadata equally and is inherently more reliable. By applying the company’s Information Dispersal technology to slice and disperse data, single points of failure are eliminated. As data is distributed evenly across all Slicestor nodes metadata can scale linearly and infinitely as new nodes are added, thus reducing any scalability bottlenecks and increasing performance. This approach delivers the combination of analytics and storage in a geographically distributed single system allowing organizations to scale their big data environments to hundreds of petabytes and even exabytes today.

"There isn’t an industry today that’s untouched by big data or a company that wouldn’t benefit from the intrinsic value of that data if they could collect, organize, store and analyze it in a cost-effective manner," said John Webster, senior partner at Evaluator Group.

"Cleversafe’s approach to combining dispersed storage and Hadoop for analytics is a groundbreaking step for the industry and for any company to effectively bridge storage and large-scale computation," said Webster.

No market segment has a more critical need to harness big data than the Government sector. Lockheed Martin Corp. is partnering with Cleversafe to develop a federal version of the Dispersed Compute Storage solution designed for the unique needs of federal government agencies.

"By combining the power of Hadoop analytics with Cleversafe’s Object-based Dispersed Storage solution, government entities will be able to significantly reduce their total cost of infrastructure as the amount of their mission critical data grows," said Tom Gordon, CTO and VP of engineering of Lockheed Martin’s Information Systems and Global Solutions-National.

"The Federal community has been out in front of big data, well ahead of many other market segments, and needs technology solutions today that are well suited for Exabyte scale storage as well as massive computation," said Gordon. "Taken Cleversafe’s approach with Hadoop across commodity hardware, these features deliver a new approach to bring the true potential of big data analytics into reach."

Cleversafe’s object-based storage solution is 100 million times more reliable than traditional RAID-based systems and it doesn’t rely on replication to protect information. Its information dispersal capabilities reduce storage costs up to 90% while meeting compliance requirements and ensuring protection against data loss, whether it’s latent hardware errors, data corruption or malicious threats. With the combination of limitless scale, reliable storage and efficient analytics in the same platform, Cleversafe is solving the most challenging big data problems for customers in an efficient manner.