AWS Announces Snowball Appliances and Kinesis Firehose

Amazon Web Services, Inc. (AWS), an Amazon.com company, announced two capabilities to help customers more quickly and cost-effectively transfer data of all types and sizes to the AWS Cloud.

Snowball appliance

Snowball is a petabyte-scale data transport appliance that can securely transfer 50TB per appliance of data into and out of AWS. Kinesis Firehose is a fully managed service for loading streaming data into AWS (available today for Amazon S3 and Redshift, with other AWS data stores coming soon).

Customers are moving an increasing number of applications and large volumes of data to the AWS cloud – everything from app log files to digital media, genomes, and petabytes of sensor data from connected devices. While AWS Direct Connect provides customers with a dedicated, fast connection to the AWS network, Snowball and Kinesis Firehose are ideal for customers that need to transfer data in large batches, have data located in distributed locations, or require continuous loading of streaming data. With Snowball and Kinesis Firehose, AWS customers have two ways to easily and cost-effectively get large data sets and streaming data into the AWS Cloud.

“It has never been easier or more cost-effective for companies to collect, store, analyze, and share data than it is today with the AWS Cloud,” said Bill Vass, VP, AWS storage services. “As customers have realized that their data contains key insights that can lead to competitive advantage, they’re looking to get as much data into AWS as quickly as possible. AWS Snowball and Amazon Kinesis Firehose give customers two more important tools to get their data into AWS.“

Snowball: fast way to transfer large amounts of data to AWS securely
Customers who need to transfer large amounts of data to AWS face a challenge – the time it takes to upload data. For instance, if a company committed 100Mb/s of their total bandwidth capacity to transferring data to AWS, transferring 100TB of data via that connection would take about 100 days. Companies could choose to spend more money expanding their bandwidth capability or upgrading their network, but most don’t want to do so simply to support sending more data to the cloud. Now, AWS provides a solution to this problem with AWS Snowball – a durable and tamper-resistant, encrypted, and portable storage appliance that customers can use to move that same 100TB of data to AWS in less than a week, and at as little as one-fifth of the cost of using high-speed Internet. Customers create a job using the AWS Management Console, AWS ships the appliance directly to the customer, and the customer, upon receiving the appliance, simply plugs it into their local network. Snowball provides a simple data transfer client which customers use to encrypt and transfer 50TB of data to each appliance. Customers can use multiple Snowball appliances in parallel to transfer larger data sets within the same time frame. Once a customer’s data is completely loaded onto an Snowball, its E Ink shipping label is automatically updated with the AWS shipping address, and customers can track the status of the transfer job using Amazon Simple Notification Service (SNS), text messages, or the AWS Management Console.

Click to enlarge

Snowball appliances use multiple layers of security to protect customer data. In addition to tamper-resistant enclosures, AWS Snowball also employs end-to-end, 256-bit encryption, along with an industry-standard Trusted Platform Module (TPM) designed to ensure both security and full chain-of-custody for customer data. Once a customer’s data has been transferred from the Snowball to AWS’s data stores (initially S3), AWS erases all data from the Snowball appliance, following the standards defined by the National Institute of Standards and Technology (NIST) guidelines for media sanitization. Customers can get a report and confirm that all of their data has successfully been loaded into AWS before deleting the local copy of their data.

Click to enlarge

BuzzFeed, Inc. is a global media company that produces and distributes original news, entertainment, and video to over 200 million unique monthly visitors and over 1.5 billion viewers.

“As one of the world’s largest media companies, Buzzfeed generates a massive amount of content that we need to preserve and maintain,” said Eugene Ventimiglia, director, technical operations, Buzzfeed. “We’ve been wanting to implement an archive solution based on Amazon Glacier, but our 250TB of archive data was too much data to migrate on individual disk drives, and using high-speed Internet would simply take too long. The release of AWS Snowball completely changes our thinking here as we now have a simple, secure, and cost-effective way to migrate the entire 250TB library to AWS in less than a week.“

Sony’s Ci Media cloud platform provides contribution, collaboration, and transformation to professional media workflows.

“Many of our customers have petabytes of data and limited bandwidth which has been a barrier for them to moving some of their heaviest media workflows into Sony Media cloud (Ci),” said David Rosen, VP, solutions business development, Sony Professional Solutions Americas. “AWS Snowball has the potential to solve this problem for us in a big way. From our early look at AWS Snowball, we are very excited at the prospect of a secure and cost-effective alternative to Internet transfers of the massive amount of media assets we have into AWS.“

Click to enlarge

Kinesis Firehose: Easily load streaming data to AWS
Mobile devices, software applications and services, wearables, industrial sensors, and IT infrastructure generate staggering amounts of data – sometimes TBs per hour. In 2013, AWS introduced Kinesis Streams to allow customers to build applications that collect, process, and analyze this streaming data with very high throughput. Many customers use Kinesis Streams to capture streaming data and load it into S3 or Redshift. Until now, this required customers to manage the Kinesis data streams and write custom code to load the data. Now, Kinesis Firehose makes this as easy as an API call. Kinesis Firehose captures data from hundreds of thousands of different sources and loads it directly into AWS, in real-time. Customers simply create an Kinesis Firehose Delivery Stream in the AWS Management Console and specify the target S3 bucket or Redshift table, and the time frequency at which they want fresh data delivered to the destination. Customers can also configure Kinesis Firehose to batch, compress, and encrypt streaming data before delivery at specified time intervals.

Click to enlarge

Monitoring Kinesis Firehose delivery streams
Click to enlarge

Hearst Corporation is one media and information companies in the world, with over 250 sites worldwide.

“We monitor trending content on our digital properties, and generate terabytes of streaming data every day,” said Rick McFarland, VP, data services, Hearst. “This is the data currency of our company and feeds our business intelligence and product development. We are excited that we can just point our fleet of web servers at Amazon Kinesis Firehose, and it takes care of all aggregation, compression, and delivery of our streaming data to Amazon S3 without any intervention from us. This will reduce operational complexity from our data pipeline and allow us to focus on analytics so we can provide the best content to our customers.“

Snowball and Kinesis Firehose are both available.

Kinesis Firehose is initially available in the US East (N. Virginia), US West (Oregon), and EU (Ireland) Regions, and will expand to additional regions in the coming months.