Deutscher Wetterdienst Prepares for Weather and Climate Data Influx

Deutscher Wetterdienst (DWD) monitors weather and meteorological conditions across Germany.

In addition to providing weather services for the general public, DWD collaborates with national and worldwide environmental research institutes and organizations to assist in solving global challenges, such as identifying alternative energy sources, maintaining air quality, and improving disaster and water management.

DWD also provides a range of climatology services, including analytical and evaluation services, expert reports, specialized forecasts, and it provides weather information systems.

Each year, DWD generates 90,000 weather forecasts, 20,000 warnings and 500,000 aviation meteorological reports, based on data from 2,000 national weather stations and observational data from all over the world.

DWD’s headquarters collects measured weather data from meteorological information systems across Germany. Each day, its data systems receive 30TB of new data, which must be accessible at all times to accurately predict weather events and provide information for science and research. A long-time user of StorageTek tape drives, DWD needed a reliable storage system to expand its infrastructure so it could handle and manage its massive data influx.

To meet its goals, DWD deployed Oracle Corp.‘s StorageTek modular library systems and tape drives and StorageTek Automated Cartridge System Library Software (ACSLS), as well as Oracle’s SPARC Enterprise servers and Oracle Solaris. These systems store forecast data and the data DWD collects from other weather stations.

The flexibility of the StorageTek library system enabled DWD to replace older components with newer modules while continuing to operate, without interrupting the system’s ability to handle incoming weather data. Further, DWD optimized the storage system’s performance by running StorageTek Automated Cartridge System Library Software on Sun SPARC Enterprise T5120 servers with Solaris 10.

"Oracle’s StorageTek SL8500 is a rock-solid library system with complete reliability that helps us to store more weather data and achieve better availability. We are very confident with Oracle’s solution, because it simply does what it is supposed to do," said Dr. Henning Weber, Data Center Chief, Deutscher Wetterdienst.

Challenges

• Ensure that the organization can buffer and integrate 30TB of incoming weather and climate data daily during the storage and server implementation phase to ensure business continuity
• Protect and ensure access to existing weather and climate data and investments in tape drives, while simultaneously moving the massive data stores to the new library system
• Deploy a scalable storage solution that can scale to accommodate increases in data volume – which the organization projects will grow from 30TB to 100TB daily – to keep up with new developments in weather-observation satellites and HPC technology
• Ensure minimal downtime during migration, as DWD’s data center needs to remain operational 24/7 to provide weather forecast data to airlines, shipping, and to other industries
• Minimize the physical infrastructure footprint, as data center space is limited
• Provide for interoperability between the existing solution and new modules, to minimize effort and ensure a smooth transition

Solutions

• Implemented two StorageTek SL8500 modular library systems from Oracle in five days, thanks to the system’s modular architecture, establishing a fast, reliable, and future-proof storage system for DWD’s weather data
• Achieved on-the-fly upgrades to the tape drives without interruption to data storage operations, by replacing the older tape drives, step-by-step, with new-generation, StorageTek T10000C tape drives to take advantage of their 5TB uncompressed capacity and throughput performance
• Saved US$1.3 million due to the new system’s backward compatibility, enabling DWD to transfer its 13,000 existing tape cartridges into the new library modules instead of purchasing replacement tape cartridges
• Ensured a data migration to the new tape library, since the new systems from StorageTek did not require retraining to use
• Increased data center capacity to store 20,000 enterprise tape cartridges – capable of archiving up to 100TB of data per day – while using just a few square meters of data center space
• Upgraded the tape library to handle up to 100PB of data, a 400% increase in storage capacity over the previous tape library, which could only store up to 20PB of data
• Adopted StorageTek SL8500 to facilitate integration with DWD’s existing stores of meteorological data
• Achieved quick data access and high system performance, thanks to SPARC Enterprise T5120 running Solaris 10
• Accomplished migration to StorageTek SL8500 – which included moving to a new office building as well as moving 13,000 tape cartridges with weather, forecast, and research data-in just one week
• Provided external and internal scientific research teams with fast and easy access to a well-organized archive holding more than 40 years of data collected from weather stations around the globe

As a long-time StorageTek user, DWD chose Oracle because it could integrate its existing system with a new StorageTek environment. It selected Solaris 10 because the system’s modules are tailored to work with StorageTek SL8500’s components, ensuring reliability and performance since all the organization’s data is maintained in a single source.
 
"For our highly-demanding data storage needs, we use proven and tested Oracle technology. We believe that the StorageTek SL8500 is a modular and expandable library system that represents the best technology available in the market," said Dr. Henning Weber, head of system support, Deutscher Wetterdienst.

Furthermore, StorageTek tape and library solutions help DWD to reduce costs. Due to the new system’s backwards compatibility, DWD can continue using its 13,000 existing tape cartridges in the new library modules, saving the organization $1.3 million in replacement costs. If DWD had chosen another vendor, it would also have had to copy the entire existing data stock to the new system. This process would have taken approximately 12 months and required additional resources, since the old and new systems would have had to run side-by-side during the migration to avoid interrupting key weather data storage operations.

The StorageTek tape and library solution also delivers storage density and requires a small amount of data center space. Because the size of DWD’s data center is limited, the new solution had to be large in storage capacity but physically small. Other solutions did not meet DWD’s capacity requirements within the restrictions on the physical footprint.