Some SAS SSDs From HPE and Other Companies (Dell EMC, WD?) to Fail

This document (ID: a00097382en_us) is a support communication – customer bulletin
published by Hewlett Packard Enterprise Development LP on March 20, 2020.

Document Version: 1
Bulletin: HPE SAS Solid State Drives – Critical Firmware Upgrade Required for Certain HPE SAS Solid State Drive Models to Prevent Drive Failure at 40,000 Hours of Operation
Notice: The information in this document, including products and software versions, is current as of the release date. This document is subject to change without notice.

Description
Important: While the failure mode is similar, this issue is unrelated to the SSD issue detailed in this customer bulletinhttps://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00092491en_us released in November 2019, which describes an SSD failure at 32,768 hours of operation.

In addition, this issue is not unique to HPE and potentially affects other manufacturers and customers that purchased these drives.

Important: This HPD7 firmware is considered a critical fix and is required to address the issue detailed below. HPE recommends immediate application of this critical fix. Neglecting to update to SSD Firmware Version HPD7 will result in drive failure and data loss at 40,000 hours of operation and require restoration of data from backup if there is no fault tolerance, such as RAID-0 or even in a fault tolerance RAID mode if more SSDs fail than can be supported by the fault tolerance of the RAID mode on the logical drive. Example: RAID-5 logical drive with 2 failed SSDs.

HPE was notified by a SSD manufacturer of a firmware defect affecting certain SAS SSD models (reference the table below) used in a number of HPE server and storage products (i.e., HPE ProLiant, Synergy, Apollo 4200, Synergy Storage Modules, D3000 Storage Enclosure, StoreEasy 1000 Storage)*. The issue affects SSDs with an HPE firmware version prior to HPD7 that results in SSD failure at 40,000 hours of operation (i.e., 4 years, 206 days, 16 hours).

Important: Due to the SSD failure not occurring until attaining 40,000 hours of operation and based on the dates these drives began shipping from HPE, these drives are not susceptible to failure until October 2020 at the earliest.

Note: The following platforms are not affected by this issue: 3PAR StoreServ Storage, D6000/D8000 Disk Enclosures, ConvergedSystem 300/500, MSA Storage, Nimble Storage, Primera Storage, SimpliVity, StoreOnce, StoreVirtual 4000/3200 Storage, StoreEasy 3000 Storage, XP Storage, SAP HANA.

After the SSD failure occurs, neither the SSD nor the data can be recovered. In addition, SSDs which were put into service at the same time will likely fail nearly simultaneously. Restoration of data from backup will be required in non-fault tolerance modes (e.g., RAID-0) and in fault tolerance RAID mode if more drives fail than what is supported by the fault tolerance RAID mode logical drive.

To determine total power-on hours via Smart Storage Administrator, refer to Smart Storage Administrator (SSA) – Quick Guide to Determine SSD Uptime

In addition, scripts which perform an SSD firmware check for the 40,000 power-on-hours failure issue on certain HPE SAS SSD drives are available for Linux, VMware and Windows. See the Readme.txt file contained within the zip file for instructions. Download the script at SSD Issue Detection Scripts

For more details, please refer to the drive firmware release notes for the version with this fix listed in the resolution section.

Scope
Any of the following HPE SAS SSD models running a drive firmware version prior to HPD7:

Hpe Sas Ssds Fail

Note: This issue is not unique to HPE and potentially affects all customers that purchased these drives.

In addition, drives contained within SAN-attached or other external storage sources must be checked separately from the drives in the main system.

Note: The list of impacted SSDs identified in this documentation is comprehensive. No other drive models are affected.

Resolution
Immediately upgrade the drive firmware to version HPD7, which HPE has released to prevent the issue described above.

Note: HPE recommends performing an online firmware update on HPE Gen9 servers during minimal I/O activity. This will not require a reboot in most circumstances. However, in instances where the online firmware update does not complete successfully, an offline update is required. After the flash update completes, the Smart Component will provide a message regarding whether the flash completed successfully.

Important: A reboot is required when updating HPE Gen8 servers. In addition, online firmware flashing of drives attached to a HPE Smart Array controller running in Zero Memory (ZM) mode or HBA is not supported. Only offline firmware flashing of drives is supported for these configurations.

HPD7 firmware for EK0800JVYPN, EO1600JVYPP, MK0800JVYPQ and MO1600JVYPR drives:

Alternatively, HPD7 drive firmware can be obtained by navigating the HPE Support Center as in the following example:
1. Go to the HPE Support Center
2. Enter a product name (e.g., EK0800JVYPN) in the text search field and wait for a list of suggested products to display. From this list, identify the desired product and select it.
3. The page should refresh to display the “Drivers and Software” tab and the components that support the selected product. From the “Drivers and Software” expandable filter menus on the left side of the page:

Select the Software Type – Firmware
Select the Software Sub Type – SAS Storage Disk
For further filtering if needed – Select the specific operating system from the operating environment.

4. Locate and select ** Critical ** Online HDD/SDD Flash Component for EK0800JVYPN, EO1600JVYPP, MK0800JVYPQ and MO1600JVYPR Drives version HPD7 (or later)

Note: To ensure that you have selected the latest version of the firmware/driver, click the Revision History tab to check if a new version of the firmware/driver is available.
For more important information, review the release notes tab.

5. Click the download button.
Note: For Storage platforms potentially affected, follow the same Resolution and apply SSD firmware HPD7 on the HPE SAS SSDs installed in the Storage solution.

Firmware Release Notes
Issue Fixed

The issue affects SSDs with an HPE firmware version prior to HPD7 that results in SSD failure at 40,000 hours of operation (i.e., 4 years, 206 days 16 hours). Neither the SSD nor the data can be recovered after the SSD failure occurs.
In addition, SSDs which were put into service at the same time will likely fail nearly simultaneously.

For HPD7 installation instructions, refer to the Installation Instructions tab on the download page for each Smart Component.

Refer to the following resources for more details on the SSD firmware update completion messages that may be displayed by the Smart Component during the update and for information on the Software Analysis and Update Tools available:

* Hardware platforms affected: HPE ProLiant ML30 Gen9 Server, HPE ProLiant DL20 Gen9 Server, HPE Synergy 480 Gen9 Compute Module, HPE Synergy 660 Gen9 Compute Module, HPE Synergy 620 Gen9 Compute Module, HPE Synergy 680 Gen9 Compute Module, HPE Apollo 6500 Gen9 System, HPE ProLiant XL270d Gen9 Server, JBOD disk, D3700 disk, HPE Apollo 2000 System, HPE 800GB SAS 12G Write Intensive 3yr Wty EK0800JVYPN Solid State Drive, HPE 1.6TB SAS 12G Write Intensive 3yr Wty EO1600JVYPP Solid State Drive, HPE 800GB SAS 12G Mixed Use 3yr Wty MK0800JVYPQ Solid State Drive, HPE 1.6TB SAS 12G Mixed Use 3yr Wty MO1600JVYPR Solid State Drive, HPE StoreEasy 3000 Gateway Storage Blade, HPE StoreEasy 1000 Storage, HPE D2220sb Storage Blade, HPE D3000 Disk Enclosures, HPE ProLiant DL160 Gen9 Server, HPE ProLiant DL180 Gen9 Server, HPE ProLiant DL360 Gen9 Server, HPE ProLiant BL460c Gen9 Server Blade, HPE ProLiant DL380 Gen9 Server, HPE ProLiant ML350 Gen9 Server, HPE ProLiant XL230a Gen9 Server, HPE ProLiant XL250a Gen9 Server, HPE ProLiant DL388 Gen9 Server, HPE ProLiant DL120 Gen9 Server, HPE ProLiant ML110 Gen9 Server, HPE ProLiant XL170r Gen9 Server, HPE ProLiant XL190r Gen9 Server, HPE ProLiant WS460c Gen9 Graphics Server Blade, HPE ProLiant DL580 Gen9 Server, HPE ProLiant BL660c Gen9 Server Blade, HPE ProLiant DL560 Gen9 Server, HPE Apollo 4200 Gen9 Server