R&D: Implementation of Storage and Analytics Platform for Big Data Lake of Electricity Usage With Spark

The Journal of Supercomputing has published an article written by Chao-Tung Yang, Department of Computer Science, Tunghai University, Taichung City, 407224, Taiwan, ROC, and Research Center for Smart Sustainable Circular Economy, Tunghai University, No. 1727, Sec.4, Taiwan Boulevard, Taichung City, 407224, Taiwan, ROC, and Research Center for Nanotechnology, Tunghai University, No. 1727, Sec.4, Taiwan Boulevard, Taichung City, 407224, Taiwan, ROC, Tzu-Yang Chen, Department of Computer Science, Tunghai University, Taichung City, 407224, Taiwan, ROC, Endah Kristiani, Department of Industrial Engineering and Enterprise Information, Tunghai University, Taichung City, 407224, Taiwan, ROC, and Department of Informatics, Krida Wacana Christian University, Jakarta, 11470, Indonesia, and Shyhtsun Felix Wu, Department of Computer Science, University of California, Davis, CA, 95616, USA.

Abstract: “Electricity data could generate a large number of records from smart meter day by day. The traditional architecture might not properly handle the increasingly dynamic data that need flexibility. For effective storing and analytics, efficient architecture is needed to provide much greater data volumes and varieties. In this paper, we proposed the architecture of data storage and analytic in the big data lake of electricity usage using Spark. Apache Sqoop was used to migrate historical data to Apache Hive for processing from an existing system. Apache Kafka was used as the input source for Spark to stream data to Apache HBase to ensure the integrity of the streaming data. In order to integrate the data, we use the Hive and HBase principle of Data Lake as search engines for Hive and HBase. Apache Impala and Apache Phoenix are used separately. This work also analyzes electricity usage and power failure with Apache Spark. All of the visualizations of this project are presented in Apache Superset. Moreover, the usage prediction comparison is presented using HoltWinters algorithm.“