Report on Library of Congress/San Diego Supercomputer Center Data Transfer and Storage Tests

The Library of Congress has published Data Center for Library of Congress Digital Holdings: A Pilot Project; Final Report.

Here an excerpt from the "Introduction":

Between May 2006 and October 2007, the Library of Congress (LC) and the San Diego Supercomputer Center (SDSC) conducted data-transfer and storage tests. At the heart of the project was the issue of trust, specifically how the LC could trust SDSC to reliably store several terabytes of the LC’s data. By what means could SDSC prove to the LC that the data was intact, preserved, and well-cared for? What tests could the LC devise, and what metrics could SDSC produce, to guarantee the integrity of their remotely stored data?

The two main objectives of the project were:

  • For SDSC to host LC content reliably and return it intact at the end of the project
  • For LC to be able to remotely access, process, analyze, and manage that content . . . .

Inspired by SDSC’s staggering technological potential, the LC had devised several scenarios for the data tests. But ultimately, as the project progressed, the LC opted to keep its goals simple: data transfer, storage, and file manipulation. In the end, both partners were happy with the project’s success. The project also produced lessons and unexpected results, some of which will have deep implications for all cultural institutions regarding transfer and storage of their digital assets.