Harvesting Field Station Data: Automating Data Flow from Raspberry Pi Sensors to Collaborative Websites

Abstract

Field stations increasingly leverage remote sensors for large scale environmental data collection. Here we demonstrate a proof-of-concept workflow from data collection from remote sensors to presentation of summary results on a remote - and therefore fast and stable - cloud server. Environmental data is collected via raspberry pis in several locations and the data is streamed to the server on XSEDE's Jetstream, housed in part at Indiana University, through low-bandwith messaging. The Jetstream cloud server does all the heavy lifting, exporting the data into a database, running automatically updating summary scripts to produce graphs, and hosting a Drupal-based website to present the data to collaborators or the public. While we use compact data in our demo, larger databases can be backed up on XSEDE's Wrangler, a large scale storage server also housed in part at Indiana University.

The end product is automatic aggregation and back up of sensor data onto a stable website that does not require a in-house server or large bandwidth on-site. This workflow is packaged into a ready-to-use and publically-available Jetstream image, meaning researchers could use their own sensors and R code for custom graphs with very little set up. Alternatively, the image can be used to house and display larger scale databases from other data types, such as audio recordings or photography. Future work will be in developing the ability to "pick up" data via drone fly-over and aggregation of citizen science data from multiple sites.