Configuring HPE Primary Storage on-Prem Real-Time Performance Monitoring with Grafana
Solution can be deployed on either Kubernetes cluster as services running inside containers, or on any type of Linux machine as few background services.
This is a Press Release edited by StorageNewsletter.com on November 10, 2022 at 2:02 pm
By Iris Wagner, senior software engineer, data infrastructure TME team, HPE Storage
In my first blog in this 2-part series, I introduced to a solution that brings you an on-prem and multi-array real-time performance visualization experience. Now, let’s dive into the technical details of how to configure your own environment to start the real-time monitoring of your HPE storage arrays.
In my first blog on this topic, I talked about real-time performance monitoring with HPE Storage Array Exporter, Prometheus, and Grafana. Now I’m sure you’re ready to really dig in and get started on configuring your own environment so you can begin monitoring your HPE storage arrays.
Let’s get started
This solution can be deployed on either a Kubernetes cluster as services running inside containers, or on any type of pf Linux machine as a few background services. For the purpose of this discussion, the deployment I’m showing is with a Linux CentOS VM. However, if you choose to deploy your solution on a Kubernetes cluster, the documentation is readily available – and know that the inner workings and most of the configuration remain very similar (if not the same), regardless of the deployment environment. Refer to the HPE Storage Array Exporter GitHub page for exporter in Kubernetes deployment, and Prometheus and Grafana websites for documentation on how to deploy each service in a Kubernetes environment.
Gathering data – Storage Array Exporter
The deployment is as easy as running a couple of commands. First download the binary of the exporter from the GitHub releases page, and change the executable file to have the appropriate access, then your exporter will be ready to run. Remember to either accept the EULA in the run command or as a system variable.
Even though each Storage Array Exporter instance only pulls metrics from one storage array, you can run multiple instances of the service simultaneously. The default endpoint the exporter pushes the metrics to is at 127.0.0.1:8080 or localhost:8080, therefore when you are running multiple instances of the service, make sure each service is pushing to a unique endpoint to avoid clashing, for example localhost:8081 for the 2nd instance.
Once your Exporter instances are up and running, you can visually examine those by open a browser on your local machine, and navigate to the endpoints your instances are pushing to, for example, the default address to browse the raw data is http://localhost:8080/metrics. I found this feature to be particularly helpful for troubleshooting.
Video: Grafana Exporter Watch video for run through of installation
Neat and organized – Prometheus
Once you have the data, you just need a service to organize the data together in format ready to be visualized. Prometheus is here for the job. You can go directly to the Download page of Prometheus website, and choose the tar ball appropriate to your environment. Download and extract the package. Before running the service, you’ll want to configure some parameters important for the solution to interface smoothly with others in the configuration yaml file.
First, you need to add some relabeling configuration copied from any of the HPE Storage published Dashboard page. This is to normalize the source metrics when they could come from different arrays that use different terminologies for storage-related labels (such as 3PAR, HPE Primera, and Alletra 9K use ‘CPG’). For a similar concept, Nimble and Alletra 6K arrays use ‘pool’ instead. Thus the labeling configuration makes it possible to view all those different types of arrays in the same Grafana dashboard.
Secondly, you have to configure Prometheus to point to the exporter endpoints as the Prometheus targets. In this same step, you can also label each target a name that makes it easier for you to distinguish between multiple arrays, such as model name or serial number.
Lastly, we can specify the scrape intervals and scrape timeouts. Be mindful that the scrape interval does not determine the time resolution of your metrics. That is determined by how often the arrays are collecting the data locally, which by default is every 5mn. In other words, specifying the scrape interval to be anything more frequent than 5mn will not make any difference in the performance monitoring results.
Finally, you can start running the Prometheus service. Once the service is running, Prometheus provides its own web UI that you can access by navigating to the default endpoint at localhost:9090 on any browser. Once verified your targets are in a healthy state, you can browse or even graph any metric provided by the targets from the Prometheus web UI.
Watch this short video for a quick walk through of the Prometheus deployment and installation process.
Grafana dashboard
Click to enlarge
Beautiful graphs – Grafana
Here comes the final stretch. You’re now ready to install and configure Grafana to display all the metrics from Prometheus in a graphical format. Download and install Grafana rpm from the Grafana website. Start Grafana as a service. You can then access the web UI through the default endpoint at localhost:3000. You can choose to modify the default login credentials from admin/admin or you can modify it directly through the web UI after you log in. Make sure your browser is on version supported by Grafana.
At this point, if your Prometheus instance is up and running, go right ahead and configure Grafana to use your Prometheus instance as its data source. The last thing left to do is to import any or all of the HPE Storage published Dashboards JSON file as a new dashboard in Grafana.
Short demo video that runs through the installation and configuration process : Grafana configuration
Make it mine – customization
You could stop here and start real-time performance monitoring as is. But what if you want your dashboards to be tailored so you can view metrics that are only important to you? Grafana is an open source project, which makes it flexible and easy to modify to fit your needs.
In the following video, I’ll show you how I added a dashboard variable volume. With the filtering capability, the graphs can show the metrics from the volume(s) selected. Moreover, you can create a new graph that plots IO/s showing only the selected volume(s). You can follow along to customize your dashboard the same way, but more importantly, I hope this example will inspire you to get creative to make your own dashboards that best serve you.
My goal with my blogs has been to show you a different way of monitoring your data center. In addition to all the other amazing tools, I hope you give this solution a try. As you get to know more about the HPE Storage Array Exporter for Prometheus and want more features or metrics, just let me know. I’m connected with a whole community ready to support you.
Resources:
Sign up for HPE Developer Community
Sign up to HPE DEV on Slack, or just simply drop a note in GitHub.
Read first blog on this topic where I talk more about HPE Primary Storage on-prem real-time performance monitoring with Grafana.
Subscribe to our free daily newsletter
