IDC Report on WW Object-Based Storage in 2016

IDC MarketScape: Worldwide Object-Based Storage 2016 Vendor Assessment

By Amita Potnis, research manager, storage systems program, IDC Corp.

IDC MarketScape WW Object-Based Storage

(Source: IDC, 2016)

IDC Opinion
The storage market has come a long way in terms of understanding object-based storage (OBS) technology and actively adopting it. The OBS landscape now sees a standardization and integration of the Amazon S3 and OpenStack Swift API, an emphasis on computing paradigms such as containers, native support for file interfaces, and a focus on vertical/use case-centric solution offerings. Many OBS solutions offer a scale-out file system that integrates with an independent but underlying OBS platform. These advancements have opened the doors for OBS so as to cater to a wide range of use cases beyond backup and archive. OBS solutions serve newer use cases that include (but are not limited to) mobility, analytics, compliance, and commercial high-performance computing areas such as media rendering/production. Today, OBS platforms are available in a variety of delivery models including appliances, software-only, and cloud services offerings.
 
According to Worldwide File- and Object-Based Storage Forecast, 2016-2020 (IDC #US41685816, September 2016), object-based storage capacity is expected to grow at a CAGR of 30.7% from 2016 to 2020, reaching 293.7EB in 2020. In terms of revenue, IDC expects the OBS market to reach $19.8 billion in 2020, much of which is driven by hyperscale environments.

In this report, IDC assesses the present commercial OBS supplier (suppliers that deliver software-defined OBS solutions as software or appliances much like other storage platforms) landscape. Cloud-based storage services based on OBS are not included in this IDC MarketScape. Since the publication of the last report on OBS, the landscape has changed dramatically – a fragmented OBS market has now somewhat consolidated. As demand for traditional external storage systems continues to decline, several mainstream storage suppliers enhanced their portfolio to include OBS solutions by developing them in-house or via acquisitions. This study assesses 12 OBS suppliers that are ‘owners of IP.’

Key findings include:

• End users and suppliers are exploring the idea of all-flash array OBS solutions that will cater to not just capacity needs but also performance requirements for environments like big data/analytics, rich media, and technical computing. Some of these solutions feature scale-out architectures based on custom flash modules (CFMs) instead of SSDs with higher densities and lower cost per gigabyte than primary all-flash arrays. Such platforms, having disrupted the primary storage market, are now targeting secondary storage markets.
• An OBS solution that supports traditional (current-generation) and next-generation applications will enable end users to adopt this technology more efficiently. For example, native support for file interfaces allows a user to write and retrieve files to/from the OBS solution.
• End users often appreciate the flexibility of choice not just in delivery models (appliances, software only, or cloud storage) but also in the ability to tier data based on usage to appropriate storage (HDD, cloud, tape, etc.). An OBS solution with strategic ISV and technology partnerships thus offering a strong solutions portfolio is also a key consideration for any end user.

IDC Marketshare Inclusion Criteria
This study assesses the capabilities and business strategies of leading suppliers in the (scale-out) OBS market segment – which is part of the overall file – and OBS market. This evaluation is based on a comprehensive framework and a set of parameters that gauge the success of a supplier in delivering an OBS solution in the market. This study includes analysis of the 12 most notable players in the commercial OBS market, with broader portfolios and global scale. The suppliers enlisted in this study are (in alphabetical order): Caringo, Cloudian, DataDirect Networks (DDN), Dell EMC, Exablox, HDS, IBM, NetApp, Red Hat, Scality, SwiftStack, and Western Digital.
 

To make this list, the suppliers need to have an OBS platform that:

• Conforms to IDC’s taxonomy on OBS platforms: software-based OBS platforms can run on any commodity x86 platforms and do not have any specific hardware customizations (like custom ASICs or SoCs) mated to the software stack, and they leverage an OBS data organization scheme.
• Has been developed in-house or owned by way of an acquisition: In other words, the supplier needs to be the IP owner of that platform.
• Is delivered as software, hardware (appliance or gateway), and/or as (private or public) cloud based. Additional points were granted to software-based platforms that can be installed on any commodity-based hardware.
• Is sold as licensed software directly to buyers or indirectly via OEM/channel partners and not just as a service. Additional points were granted if the supplier had partnerships with as-a-service providers to deliver it as a cloud offering.
• Was generally available as a current offering at the time IDC undertook this study in early 2016.
   

Note that certain suppliers (e.g., Scality, Red Hat) are pure-play software vendors, while the other suppliers sell a mix of hardware and software, mostly as hardware appliances. Pure-play software typically represents 25-50% of the total revenue, so associated server revenue is added to compare the size of the bubbles directly to the appliance vendors.

Esssential Buyer Guidance
Digital assets are the new IP and many businesses are actively trying to create new sources of revenue streams through it. For example, media streaming, the IoT, and web 2.0, are some of the ways businesses are generating revenue in today’s digitized world. IT buyers are looking for newer storage technologies that are built not just for unprecedented scale while reducing complexities and costs but also to support traditional (current-generation) and next-generation workloads. In general, these trends are driving the adoption of industry-standard computing platforms, optimized flash memory, all-flash arrays, software-defined storage, and so forth. In essence, end users are looking for a future proof technology that can sustain data growth, performance, and capacity scaling demands at affordable costs.

According to IDC’s 2016 end-user adoption trends of object-based storage, improved provisioning times, reduced capex, and improved flexibility are among the top 3 perceived as well as actual benefits achieved through the use of OBS solutions.

Buyers should therefore look for the following key characteristics
when evaluating OBS solutions:

• Automated ILM: Policy-driven ILM capabilities that enable management, repair, and deletion of data will help increase efficiency when managing petabyte- or exabyte-scale data sets.
• Solutions portfolio: A strong technology (hardware and software) partnership portfolio lends itself to a given OBS offering being able to support many use cases across verticals. An OBS supplier with several ISV and server-hardware partnerships provides end users with the ease of procuring and deploying solutions in a quick and efficient manner.
• Platform scalability: Scalability is not just from a hardware perspective (capacity and performance) but also from throughput, file size, and file volume perspectives. A solution appropriate for a given environment will allow each dimension to scale independently. Recent innovation around the AFA OBS solutions are enabling higher performance and capacity for next-generation workloads.
• Storage efficiency: Data reduction technologies such as erasure coding, de-dupe, or replication are a must with very large data sets. Data optimization technologies that include automated data tiering to the appropriate tier (HDD, SSD, public cloud storage, tape, etc.) give end users the added benefit of storage efficiency.
• Data management and reporting: In addition to data layout and organization that can impact performance, efficiency, and availability and a user-friendly UI that supports in-depth reporting on data, applications resource usage is beneficial to end users. A solution that supports advanced metadata, indexing, and analytics will be a key component of the infrastructure.
• Data resiliency: Resiliency capabilities (like replication and erasure coding) and the granularity with which such capabilities can be applied (i.e., whether policies can be applied at an account, container, or object level) will be important considerations. Data resiliency should also be weighed against the platforms’ CAP theorem profile.

Market Definition
IDC classifies OBS platforms as part of the scale-out file- and OBS (FOBS) market segment. It uses the classification scheme to classify newer software-based file- and object-based storage platforms.

Scale-out FOBS refers to FOBS solutions that use a distributed data placement mechanism to span multiple independent server hosts or controllers while presenting a single data access namespace. Such architectures are also called shared nothing (or shared data) architectures. Such architectures allow for flexible scalability in performance and capacity independent of each other using commodity components. Data sharing and distribution mechanisms (such as local and geographic replication and local and distributed erasure coding) account for one or more concurrent component failures. Scale-out FOBS solutions are made up of two variants: scale-out FBS solutions and scale-out OBS solutions. There are two principal differences between the two types: how data is organized, and how data is accessed.

Scale-out FBS solutions use distributed file systems with hierarchical structures to organize and store data. These structures are akin to mechanisms used by monolithic file systems, which in most cases follows a root directory (folder) and inverted tree structure. In contrast, scale-out OBS solutions use flat structures to organize data. Such structures are higher-level structures in which data is often organized using an ‘account, container, object’ approach wherein ‘objects’ are analogous to ‘files’ in FBS solutions. Accounts, containers, and objects are referenced by a metadata repository that stores and manages attributes of data stored in that structure. The level at which OBS solutions operate varies from platform to platform. Many OBS solutions operate on a per-object level (i.e., allow each object to be treated independently as far as policy management is concerned) whereas others operate at a container or account level (i.e., only allow policies to be applied at a container or account level). Several OBS solutions also leverage NoSQL databases as metadata repositories and persistent data stores (instead of storing chunks in the file systems).

Because of the need to manage objects with a comprehensive set of attributes, most OBS solutions use a different set of data interfaces than their FBS counterparts that mostly leverage NFS, SMB (CIFS), or FTP protocols. It is common for many OBS solutions to support HTTP/REST, CDMI, Amazon S3, and other object-specific interfaces.

The (scale-out) OBS market subsegment, which is part of the file and OBS market, is an example of an emerging market. In this report, IDC attempts to assess the capabilities and strategies of key vendors of OBS solutions. It expects that market forces such as fierce competition and buyer demand will accelerate the metamorphosis of this market into a mature market with only a few dominant vendors. Open source-based stacks will create an additional dimension of complexity and challenges. In all likelihood, the only survivors in this market may be vendors with robust partner ecosystems and/or vendors with commercial variants of open source platforms.

“A new digitized world demands an infrastructure that is extremely scalable and flexible in terms of delivery models and also payment options, with strong vendor strategic and research vision from the vendor side along with unprecedented economies of scale,” said Potnis. “OBS platforms hold the promise and the potential to support end users along this path of digitization. In this competitive market, vendors offering OBS platforms with the most compelling value proposition via a long-term strategy, research and development plan, and flexible delivery models will survive.”