More on Failure of HPE 3par StoreServe SAN

ATO (Australian Tax Office) systems update, Media Statement, December 16, 2016

By commissioner of taxation, Chris Jordan AO on system outages

Earlier last week, the ATO suffered its worst unplanned system outage in recent memory.

This was an extremely unusual and unfortunate event with the outage caused by a significant and unprecedented failure of storage hardware. The storage hardware was upgraded in November 2015 by Hewlett Packard Enterprise (HPE) after a lengthy and thorough selection process, and was seen to be ‘state-of-the-art’ at the time. We understand the use of this storage hardware is not unique to the ATO and is basically the same used by other large clients of HPE in Australia and across the globe.

What compounded the problem beyond the initial failure was the subsequent failure of our backup arrangements to work as planned. The failure of our backup arrangements meant that restoration and resumption of data and services has been very complex and time consuming.

Let me confirm there has been no loss or compromise of data.

The issues we have experienced this week do not relate to our overall IT capability or skills.

ATO staff and people at HPE have worked around the clock over the last week to bring systems back online progressively. Our website, the Tax Agent Portal, BAS Portal and Business Portals are up and running and we are seeing continued improvements in their functionality. We are continuing to work on the stabilisation of ATO Online services.
We are realistic that there may be some intermittent performance issues in the next couple of days as the full restoration process proceeds.

We, and HPE, will continue to work on the stability and performance of all of our systems and we will have staff working over the weekend to catch-up on any backlog of work.

All refunds that were due to issue when the system went down on Monday have now been processed and will be paid today and tomorrow. Any other refunds in the system will be fast-tracked to ensure people get these payments as quickly as possible – we have devoted additional resources to ensure the vast majority of people get their refunds on time.

I repeat a message we issued earlier in the week – no tax payers will be disadvantaged as a result of the outage. This means if you needed to make a payment or lodge a form but couldn’t, you won’t be penalised. If you have any other concerns about how the outage might have affected you, please get in touch with us on 13 28 61.

We know the outages week have caused inconvenience, especially so close to Christmas, and I apologise for that. I’d like to thank our clients and partners, in particular tax and BAS agents, for their patience and understanding.

As we return to normal operations next week, we will be fully investigating the events of the last week. There will be a comprehensive and independent review to help us answer the questions – exactly what happened, how and why, and what measures need to be taken to avoid such a situation in the future.

The review will be conducted by an independent expert who will determine the nature of the failure(s) and their root cause(s), the adequacy of backup and contingency arrangements, and the likelihood of recurrence.

The review will also consider our immediate response to the failure, how we managed business resumption processes and the effectiveness of our communication with the community. We are refining the scope of the review and our latest version is attached to this statement.

I will be doing everything I can to learn from what has happened this week and to put in place any necessary changes to minimise the risk of any recurrence. Please be assured that I am committed to providing a quality, reliable and contemporary service to the Australian community.