Lenovo Highly-Available OpenStack Enterprise Cloud Platform Practice With EasyStack

In 2015, Lenovo, (Singapore) Pte Ltd chose EasyStack to build an OpenStack-based enterprise cloud platform to carry out their ‘Internet Strategy’.

OpenStack is the foundation for cloud, and perhaps has matured in the overseas market. In China, OpenStack practices worthy of noticing often come from the relatively new category of Internet companies. Though it has long been marketed as ‘enterprise-ready’, traditional industries still tend to hold back towards OpenStack. This article aims to turn this perception around by presenting an OpenStack practice from the Lenovo, detailing their journey of transformation in both the technology and business realms to a private cloud built upon OpenStack. Although OpenStack will still be largely a carrier for internet businesses, Lenovo plans to migrate 20% of its IT system onto the cloud before the end of 2016 – taking a step forward.

Be it the traditional PC or the cellphone, technology’s evolving fast amidst this move towards mobile and social networking, and the competition’s fierce. In response to changing market dynamics, the Lenovo Group made the move of going from being product-oriented to a user-oriented strategy that can only be supported by a flexible and scalable enterprise cloud platform capable of rapid iterations. After thorough consideration and evaluation, Lenovo chose OpenStack as the basis for their enterprise cloud platform to carry out this ‘Internet Strategy’. After six months of practice, this platform has evolved into an enterprise OpenStack production environment of over 3,000 cores with data growth peaking at 10TB/day. It’s expected that 20% of the IT system will be migrated onto the cloud by the end of 2016.

Transformation and Picking the Right Cloud
In the past, internal IT at Lenovo has always been channel- and key client-oriented, with a traditional architecture consisting of IBM Power, AIX, PowerVM, DB2 and more recently, VMware virtualization. In the move towards becoming an internet company, such traditional architecture was far from being able to support the user and business volume brought by the B2C model. Cost-wise, Lenovo’s large-scale deployment of commercial solutions were reliable but complex to scale and extremely expensive.

Also, this traditional IT architecture was inadequate in terms of operational efficiency, security and compliance and unable to support Lenovo’s transition towards eCommerce and mobile business. In 2015, Lenovo’s IT entered a stage of infrastructural re-vamp, in need of using a cloud computing platform to support new businesses.

To find the right makeup for the cloud platform, Lenovo performed analyses and comparisons on mainstream x86 virtualization technologies, private cloud platforms, and public cloud platforms. After evaluating stability, usability, openness and ecosystem vitality, Lenovo deemed the OpenStack cloud platform technology able to fulfill its enterprise needs and decided to use OpenStack as the infrastructural cloud platform supporting their constant businesses innovations.

DR plans on VMs, cloud HDDs and databases were considered early on into the OpenStack architectural design to ensure prompt switch over when needed to maintain business availability.

Highly Available Architectural Design
On architectural logic, Lenovo’s Enterprise Cloud Platform managed infrastructures through a software-defined-environment, using x86 servers and 10GbE network at the base layer, alongside internet-style monitoring and maintenance solutions, while employing the OpenStack platform to perform overall resource management.

To ensure HA and improve the cloud platform’s system efficiency, Lenovo designed a physical architecture, and used capable servers with advanced configurations to make up the compute, storage network all-in-one, then using OpenStack to integrate into a single resource pool, placing compute nodes and storage nodes on the same physical node.

Two-way X3650 servers and four-way ThinkServer RQ940 server as backbones at the hardware layer. For every node there are five SSD HDDs and 12 SAS HDDs to make up the storage module. SSD not only acts as the storage buffer, but also is the storage resource pool, accessing the distributed storage through the VM to achieve HA.

Lenovo had to resolve a number of problems and overcome numerous hurdles to elevate OpenStack to the enterprise.

Compute
Here, Lenovo utilized high-density VM deployment. At the base is KVM virtualization technology, optimized in multiple way to maximize physical server performance, isolating CPU, memory and other hardware resources under the compute-storage convergent architecture. The outcome is the ability to have over 50 VMs running on every two-core CPU compute node.

In the cloud environment, it’s encouraged to achieve HA through not hardware, but solutions. Yet still there are some traditional applications that hold certain requirements to a single host server. For such applications unable to achieve HA, Lenovo used Compute HA technology to achieve HA on compute nodes, performing fault detection through various methods, migrating VMs on faulted physical machine to other available physical machines when needed. This entire process is automated, reducing as much as possible business disruptions caused by physical machine breakdowns.

Network

• Network Isolation: Using different NIC, different switch or different VLAN to isolate various networks such as stand-alone OpenStack management networks, virtual production networks, storage networks, public networks, and PXE networks, so that interferences are avoided, increasing overall bandwidth and enabling better network control.
• Multi-Public Network: Achieve network agility through multiple public networks to better manage security strategies. The Public Networks from Unicom, Telecom and at the office are some examples.
• Network and Optimization: Better integrate with the traditional data center network through the VLAN network model, then optimize its data package processing to achieve improved capability on network data pack process, bringing closer the VM bandwidth to that of the physical network.
• Dual Network Protocol Bundling and Multi Switch: Achieve HA of physical networks through dual network protocol bundling to different switches.
• Network Node HA: Achieve public network load balance, HA and performance through multiple network nodes, at which router-level Active/Standby methodology is used to achieve HA, which is ensured through independent network router monitoring services.

Storage
The Lenovo OpenStack Cloud Platform used Ceph as the unified storage backend, in which storage for Glance image mirroring, Nova VM system disk, and Cinder Cloud HDD are provided by Ceph RBD. Using Ceph’s Copy on Write function to revise OpenStack codes can deploy VMs within seconds.

With Ceph as the unified storage backend, its functionality is a key metric on whether the critical applications of an enterprise can be virtualized and cloud-ready. In a super-convergent deployment architecture where compute and storage run alongside each other, storage function optimization not only have to maximize storage capability, but also have to ensure the isolation between storage and compute resources to maintain system stability. For the IO stack below, Lenovo conducted bottom-up layer-by-layer optimization:

On Networks
Open the Jumbo frame, improve data transfer efficiency while use 10GbE to carry Ceph cluster network traffics, improving the efficiency on Ceph data replication.

On Functionality
Leverage SSD as the Ceph OSD log to improve overall cluster IO functionality, to fulfill performance demands of critical businesses ( for example the eCommerce system’s database businesses, etc.) and achieve function-cost balance. SSD is known for its low power consumption, prompt response, high IO/s, and high throughput. In the Ceph log system, these are aligned to multi-thread access; using SSD to replace mechanical HDDs can unleash SSD’s trait of random access, rapid response and high IO throughput. Appropriately optimizing IO coordination strategy and further suit it to SSD and lower overall IO latency.

Purposeful Planning
Plan the number of Ceph OSD under the super-convergent node reasonably according to VM density on the server, while assign in advance CPU and other memory resources. Cgroup, taskset and other tools can be used to perform resource isolation for QEMU-KVM and Ceph OSD

Parameter Tuning
Regarding parameter tuning for Ceph, performance can be improved by fine-tuning parameters on FileStore’s default sequence, OSD’s OP thread and others. Additional tuning can be done through performing iteration test to find the most suitable parameter for the current hardware environment.

Data HA
Regarding data HA, besides existing OpenStack data protection measures, Lenovo has planned a DR protocol for its three centers at two locations:

By employing low-latency fiber-optic cable, data can be simultaneously stored in local backup centers, and started asynchronously in long-distance centers, maximizing data security.

AD Integration
In addition, Lenovo has integrated its own business demands into the OpenStack enterprise cloud platform. As a company with tens of thousands of employees, AD activity logs are needed for authorization so that staffs won’t need to be individually set up user commands. Through customized development by part of the collaborator, Lenovo has integrated AD functions into its OpenStack enterprise cloud platform.

Overall Outcomes
Lenovo’s transformation towards being ‘internet-driven’ was able to begin after the buildup of this OpenStack enterprise cloud platform. eCommerce, big data and analytics, IM, online mobile phone support and other internet based businesses, all supported by this cloud platform. Judging from feedback from the team, the Lenovo OpenStack enterprise cloud platform is functioning as expected.

In the process of building up this OpenStack based enterprise cloud platform, Lenovo chose EasyStack, the Chinese OpenStack company to provide professional implementation and consulting services, helping to build the initial platform, fostering a number of OpenStack experts. For Lenovo, community compatibility and continuous upgrade, as well as experiences in delivering services at the enterprise are the main factors for consideration when choosing an OpenStack business partner.