It’s not a data deluge – it’s worse than that

Abstract

IU was among the many organizations that developed the phrase “data deluge” to describe the prodigious capabilities of digital instruments to produce data. A deluge calls to mind an extremely heavy rain, or maybe being drenched by a large wave. Unfortunately the situation we have is worse than that. The new capabilities of next-generation sequencing machines, digital video, and the capability of scientists to put high-output devices in remote locations makes the data issue far more challenging that it has ever been. This talk focuses on two general areas of handling data issues: wide area filesystems and movement of data across long distances; and the challenges of data management when data production rates simply exceed the capabilities of the network connecting source to analysis facilities. Examples will be drawn from use of the IU Data Capacitor, now the most widely used globally-accessible file system in the history of the TeraGrid; and field studies with data sources ranging from the Antarctic ice cap to African villages to telescopes on remote mountains. Some successes and many emerging challenges will be discussed.