Google Transfer Appliance in EU for Cloud Data Migration

From Ben Chong, product manager, Transfer Appliance, Google Cloud Platform

We’re pleased to announce that Transfer Appliance is available in beta in the EU market.

100TB Transfer Appliance

Transfer Appliance is a high-capacity server that lets you move large amounts of data to Google Cloud Platform (GCP) quickly and securely. It’s recommended if you’re moving more than 20TB of data, or data that would take more than a week to upload.

You can request a Transfer Appliance directly from your GCP console. The service will be available in beta in the EU in a 100TB configuration with total usable capacity of 200TB. And it’ll soon be available in a 480TB configuration with a total usable capacity of 1PB.

Moving HDFS clusters with Transfer Appliance
Customers have been using Transfer Appliance to move everything from audio and satellite imagery archives to geographic and wind data. One popular use case is migrating Hadoop Distributed File System (HDFS) clusters to GCP.

We see lots of users run their Apache Spark and Apache Hadoop clusters on GCP with Cloud Dataproc, a managed Spark and Hadoop service that allows you to create clusters quickly, then hand off cluster management to the service. Transfer Appliance is an easy way to migrate petabytes of data from on-premise HDFS clusters to GCP.

Earlier this year, we announced the ability to configure Transfer Appliance with one or more NFS volumes. This lets you push HDFS data to Transfer Appliance using Apache DistCp (also known as Distributed Copy) – an open source tool commonly used for intra/inter-cluster data copy. To copy HDFS data onto a Transfer Appliance, configure it with an NFS volume and mount it from the HDFS cluster. Then run DistCp with the mount point as the copy target. Once your data is copied to Transfer Appliance, ship it to us and we’ll load your data into cloud storage.

Using Transfer Appliance in production

Candour Creative’s work

EU customers such as Candour Creative, which helps their clients tell stories through films and photographs, wanted to take advantage of having their content readily available in the cloud. But Zac Crawley, director, Candour, was facing some challenges with the move.

“Multiple physical backups of our data were taking up space and becoming costly,” Crawley says. “But when we looked at our network, we figured it would take a matter of months to move the 40TB of large file data. Transfer Appliance reduced that time significantly.”

Terraloupe: Identifying rooftops and infrastructure issues
from aerial imagery with deep learning.
Click to enlarge

TerraLoupe GmbH, a provider of aerial imagery analytics, has been using Transfer Appliance to move fast-growing datasets quickly.

“At Terraloupe, we analyze aerial imagery with deep learning algorithms,” says Niklas Barkmeyer, product manager, TerraLoupe. “Depending on the geographic areas and image resolutions, these geospatial datasets are typically several terabytes large for single cities and can reach petabytes for entire countries or continents. Transfer Appliance was the fastest and most convenient way for us to get these terabytes to the cloud for analytics.”

480TB Transfer Appliance

Transfer Appliance can help you meet strict data protection requirements, if required. The entire Transfer Appliance service, from shipping to data upload to GCP, is handled in the EU, so your data won’t leave the EU. You’ll be able to choose which region the data uploads to during the signup process. Be sure to review Google Cloud’s commitments to the General Data Protection Regulation, as well as your organization’s commitment to your customers regarding their data. The data you send with Transfer Appliance is encrypted at capture. You hold the encryption keys, and you decrypt your own data as you upload it to its final cloud storage destination.

Resources:
If you are interested in migrating your HDFS data to GCP, request an appliance.
Benefits of moving HDFS to GCP.
Transfer Appliance documentation.
Storage and Data Transfer
Google Cloud Platform