MinIO AIStor Sets New Standard for AI and Data-Intensive Workloads with AWS S3 Express API Support

MinIO, Inc. announced AIStor support for Amazon’s S3 Express API, a streamlined version of the general-purpose S3 API designed to deliver maximum performance for AI and data-intensive analytics workloads.

MinIO AIStor is the 1st and only AI data storage platform to adopt Amazon’s S3 Express API and enable organizations to put all of their analytical and AI data, not just a subset, in ‘express mode’ at no extra cost.

Introduced by AWS in 2023 and delivered via the Amazon S3 Express One Zone storage class, the Amazon S3 Express API has defined a new industry standard for high performance and developer-friendly AI and analytics storage, delivering 10x faster performance compared to the S3 Standard storage class. But, in AWS, the enhanced performance of AWS S3 Express One Zone comes at over 5 times the cost of S3 Standard to store data and doesn’t come with data protection and replication capabilities. Consequently, to use it safely, customers must save a durable copy of their data in S3 standard and copy data that requires high speed access into S3 Express One Zone.

With this release, MinIO is democratizing access to and use of this powerful API for all data processing needs by coupling its dramatic performance advantages with full active/passive, DR, and batch replication support, all at no extra cost. Enterprises can continue leveraging the general purpose S3 API for raw source data requiring versioning and immutability before any processing, but can now leverage the high performance S3 Express API for all other downstream workloads and derived data sets, especially, but not limited to, high intensity data processing. This includes the majority of data lakehouse analytics with Apache Spark and Iceberg, AI data pre-processing, and AI model training and inference.

“Until now, use of S3 Express has typically been reserved for data that requires high speed access at the expense of enterprise features like replication and lifecycle management, due to its cost premium over S3 Standard,” said Garima Kapoor, co-founder and co-CEO, MinIO. “We’re changing that because we believe it can and should become the standard API for all analytical data and AI workloads. With the AIStor S3 Express API, pricing remains the same so enterprises can now put all of their analytical and AI data, not just a subset, in ‘express mode.’ This is drawing tremendous excitement from customers.”

“In most lakehouse environments, versioning is already handled by table formats like Apache Iceberg, which provide atomic operations at the file or table level,” said Sanjeev Mohan, principal analyst, SanjMo. “But when versioning is also enabled at the storage layer, it can create unnecessary duplication and complexity. Removing it from the storage layer simplifies the backend architecture, lowers CPU usage on storage servers, and helps prevent accidental storage growth and rising costs.“

AIStor’s S3 Express API support comes at no extra cost, without any request charges for GETs, PUTs and LISTs, and with key data protection and replication capabilities so it can be used for all enterprise analytical and AI data.

Key features and benefits of AIStor S3 Express API include:

• Accelerated PUT and LIST operations: delivers up to 20% faster PUT operations, lowering CPU utilization, and up to 447% faster time-to-first-byte (TTFB) LIST operations relative to AIStor general purpose S3 API. This means faster training, faster analytics, and better infrastructure utilization.
• New atomic, exclusive append operations: enables direct and safe object modification, eliminating multi-step update workflows. For example, media-broadcasting applications which add new video segments to video files can do so as they are transcoded and immediately streamed to viewers.
• Full active/passive, DR, and batch replication support: provides full performance benefits of the S3 Express API without loss of business continuity and disaster recovery (BC/DR). Synchronous replication means no data loss in the event of an outage and asynchronous replication enables data protection across the globe.
• Streamlined and simplified API behavior: eliminates redundancies and implements simplifications and guardrails to improve the developer experience as well as application resiliency, predictability and security.

As the 1st and only on-premises and private cloud AI storage platform to adopt Amazon’s S3 Express API, MinIO is reinforcing its ongoing commitment to providing full and complete S3 API compatibility.

Resource:
Blog: AIStor Adds Support for the S3 Express API