New Study From Igneous Highlights Enterprises Struggling to Manage Huge Volumes of Unstructured Data

A new study from Igneous Systems, Inc., The State of Unstructured Data Management Report (registration required), shows that North America’s data-centric enterprises are working with massive, difficult-to-manage volumes of unstructured data.

Most (59%) of the respondents report managing more than 10 billion files, with an average annual growth rate of 23%.

The research found that these enterprises are indeed struggling to effectively manage today’s influx of machine- and human-generated unstructured data.

Igneous sponsored the study, which polled 200 IT leaders in North America in data-intensive organizations from a range of industries to explore this topic.

“We are witnessing a shift from transactional workflows to data-centric workflows. Machine learning workflows, for example, start with massive amounts of data and then rely on a high-volume flow of data to fuel critical decisions,” said Kiran Bhageshpur, CEO, Igneous. “These new data-centric workflows create great value as well as tremendous amounts of unstructured data that is retained longer.”

Unstructured Data Growth from Machines and Data Workflows

82% of respondents manage 1 billion or more files and objects. In fact, a staggering 59% manage more than 10 billion files.

Study respondents also reported:

• Most of their unstructured data growth (55%) comes from machine-generated data (apps, imaging, sensor data, etc.).
• The average respondent has 7 NAS storage systems. One in three has more than 10.  One in 12 has more than 50.
• AI/ML/HPC data workflows are the dominating driver of data management and IT architecture strategies across all verticals.

To further explain how different industries work with massive data stores, Igneous assembled a panel of experts for a discussion of data enablement at scale.

According to George Crump, chief dteward, Storage Switzerland, “Unstructured data growth outweighs structured by at least ten to one, maybe more and it’s mostly machine generated.”

Data is Hugely Important to Organizations  
Survey respondents reported that data is their business’ most important asset, second only to customers and employees. Respondents ranked data as more important than brand, intellectual property or even their physical assets. In fact, 40% of a typical organization’s value comes from this data.

Key Asks for Unstructured Data Management
Since data management for databases and VM’s is better-understood and already well covered by the tools on the market, the survey focused on unstructured data management. The survey explored which aspects of unstructured data management are most valuable to organizations, and also how difficult these aspects are.

Out of ten pillars of unstructured data management, the following three were ranked as most important: first, data accessibility; second, data governance; third, data insight.

Overall, this points to a desire for better access to and understanding of unstructured data. 

“Many vendors are offering storage technologies that can handle massive growth, but it’s the insight into that data that’s lacking,” says Jeff DiNisco, CTO at P1 Technologies. “Without having the tools to actually organize, move and process that data – and process it in the long term – you can’t really do anything with it. You have to be intelligent about the way you manage data so you can do something with it decades from now.”

Interestingly, while data accessibility and data insight are among the respondents’ top three data management priorities, respondents ranked these two as among the most difficult to achieve. This explains why respondents also reported performing poorly on these two goals.  

The report includes a list of top strategic tips from Igneous, whose customers are North America’s data-intensive organizations managing billions of files and petabytes of unstructured data.