Internet Archive Uses Sun for Archiving …

Sun Microsystems, Inc. announced that the Internet Archive, one of the fastest growing digital libraries in the world, has migrated its digital archive efforts onto Sun’s open hardware and software platforms and established a new primary datacenter that will be housed at Sun’s Santa Clara, California, campus. The Archive chose Sun to help move away from its customized storage architecture into a flexible and open Sun Modular Datacenter (Sun MD) comprised of open storage technology like Solaris ZFS and low-cost industry standard Sun Fire systems.

"The Internet Archive offers long-term digital preservation to the ephemeral Internet," said Brewster Kahle, founder, the Internet Archive organization. "As more of the world’s most valuable information moves online and data grows exponentially, the Internet Archive will serve as a living history to ensure future generations can access and continue to preserve these important documents over time."

Founded in 1996 by Brewster Kahle, the Internet Archive is a non-profit organization that has built a library of Internet sites and other cultural artifacts in digital form that include moving images, live audio, audio and text formats. The Archive offers free access to researchers, historians, scholars, and the general public; and also features ‘The Wayback Machine’ — a digital time capsule that allows users to see archived versions of Web pages across time. At the end of 2008, the Internet Archive housed over three petabtyes of information, which is roughly equivalent to about 150 times the information contained in the Library of Congress. Going forward, the Archive is expected to grow at approximately 100 terabytes a month.

"We’ve worked closely with the Internet Archive to ensure the right technology platform is in place to handle and manage growing amounts of the world’s most valuable data and that it lives on for future generations," said Dave Douglas, Chief Sustainability Officer and senior vice-president of Cloud Computing, Sun Microsystems, Inc. "The combination of open storage technology innovation in a Sun Modular Datacenter is a perfect match for this organization’s mission and gives them the storage performance they need in a smaller power envelope ‘ all at a cost-effective price point."

Sun’s Open Storage Innovation as Backbone of Archive
The Internet Archive looked to Sun to help with two important technological challenges: storing massive amounts of data and ensuring this data will be preserved in the future. The non-profit organization needed a way to parse, index and physically encode exponentially greater amounts of raw data; while, at the same time protect stored resources from damage or destruction. Data degradation and maintaining accessibility of the data in unknown future formatting were among the many challenges facing this project.

Sun worked closely with the Internet Archive to design an open and cost-effective solution that met their needs. Using the Sun MD platform as the basis of their datacenter, Internet Archive made the switch from their customized hardware and storage to industry standard Sun Fire x4500 systems running Solaris 10 and Solaris ZFS. Sun also agreed to host and manage the datacenter on Sun’s Santa Clara campus and will provide the power, cooling and networking capabilities.

"No company can match the storage innovation that is coming out of Sun right now," said Kahle. "Putting Sun’s highly dense storage technology into a modular datacenter gives us the performance and efficiency we need at a low price point."

The Internet Archive wanted to take advantage of the technological developments that are going on with cloud computing, yet it was also important to maintain and own the information contained in the Archive. As one of the most open companies on the planet, Sun is currently working with the Internet Archive to explore solutions that can meet all of its requirements.

To learn more about this new project