Amazon Web Services Unveils Kinesis

At AWS re:Invent, Amazon Web Services, Inc. an Amazon.com company, announced Kinesis, a managed service for real-time processing of high-volume, streaming data.

Using the service, a customer can store and process terabytes of data an hour from hundreds of thousands of sources, making it easy to write applications that take action on real-time data such as web site click-streams, marketing and financial transactions, social media feeds, logs and metering data and location-tracking events.

Kinesis-enabled applications can power real-time dashboards, generate alerts, drive real-time business decisions such as changing pricing and advertising strategies or send data to other big data services such as Amazon S3, Amazon EMR or Redshift. It scales to support applications that process data streams of nearly any size, while replicating all data across multiple availability zones to help ensure high durability and availability.

To date, most big data processing has been done through batch-oriented approaches such as those used by Hadoop, or through database technologies such as data warehouses. Since these systems store and process data in batches, they aren’t able to support applications that need to process streaming feeds of changing data in real time.

To build applications that rely on this ‘fast-moving’ data, many companies have developed their systems or stitched together open source tools, but these are often complex to build, difficult to operate, inelastic and hard to scale and can be unreliable or lose data. This helps solve these problems by providing a fully managed service that takes care of all the ‘heavy-lifting’ for developers, providing data ingestion and storage, high data durability and the ability to scale from kilobytes to terabytes an hour.

The client library simplifies load balancing, coordination and fault tolerance, and developers can use AWS auto scaling to create elastic, high-performance Amazon EC2 processing clusters.

It also integrates with third-party products, giving developers control and freedom to choose their preferred method of data processing, including open source products.

Customers can add real-time analytics and other functionality to their applications, turning data growth into an opportunity to build advantage and innovate for their customers.

“Database and MapReduce technologies are good at handling large volumes of data,” said Terry Hanold, VP of new business initiatives, AWS. “But they are fundamentally batch-based, and struggle with enabling real-time decisions on a never-ending-and never fully complete-stream of data. Amazon Kinesis aims to fill this gap, removing many of the cost, effort and expertise barriers customers encounter with streaming data solutions, while providing the performance, durability and scale required for the largest, most advanced implementations.”

With the AWS management console, or through an API call, customers can create a data ‘stream’ that captures and stores data as it is submitted, making it available to applications, open source streaming tools, and data stores (e.g. Amazon S3, Amazon DynamoDB or Amazon Redshift). For example, a customer could use the service to build an application that extracts key metrics from log data before sending it to the Redshift data warehouse service. Increasing or decreasing the size of a stream makes capacity available ins, and customers pay only for the size of their data stream.

The NASDAQ OMX Group is the world’s largest exchange company, owning and operating 24 markets across six continents, including The NASDAQ Stock Market.

“Exchange order data peaks at tens to hundreds of MBs per second, and we have to accurately record and process every single transaction,” said Ann Neidenbach, SVP of global software development, NASDAQ OMX. “Amazon Kinesis is designed to handle market order data in real-time, durably and reliably, without data loss, enabling continuous analysis and auditing of this stream of market data. This lets us focus on building the next generation of innovative financial tools and services.“