Decommissioning a Large Data Archive: Lessons Learned from Cleaning out the Attic

Abstract

This paper describes key elements of the decommissioning of a large tape-based data archive that the San Diego Supercomputer Center (SDSC) operated for its users from the center’s inception in 1985 until ~2010. This 25-year period covered many generations of supercomputers and correspondingly many generations of tape and storage technologies, with Moore’s-law growth in supercomputing power and associated storage capacity/bandwidth. Over the archive’s last decade, data volume grew exponentially with a doubling period of ~16 months to a maximum size of ~10 PB. In ~2010, the National Science Foundation terminated funding for SDSC’s tape archive and SDSC proceeded with decommissioning the archive over a ~2-year period. This paper briefly describes the principles and process by which we decommissioned this large archive, key issues that arose during this process, and implications for institutions that operate data archival systems and suggestions for operating archival systems in the FAIR data environment.