Evolution of the Australian Geoscience Data Cube

The Australian Geoscience Data Cube (AGDC) Programme envisions a Digital Earth, composed of observations of the Earth¿s oceans, surface and subsurface taken through space and time stored in a high performance computing environment. The AGDC will allow governments, scientists and the public to monitor, analyse, and project the state of the Earth. The AGDC will also realise the full value of large Earth observation datasets by allowing rapid and repeatable continental-scale analyses of Earth properties through time and space. At its core, the AGDC is an indexing system which supports parallel processing on HPC. One of the key features of the AGDC approach is that all of the observations (pixels) in the input data are retained for analysis¿ the data are not mosaicked, binned, or filtered in any way and the source data for any pixel can be traced through the metadata. The AGDC provides a common analytical platform on which researchers can complete complex full depth analyses of the processed archive (~500TB) in a matter of hours. As with the European Space Agency¿s (ESA) GRID Processing on Demand (GPOD) system (https://gpod.eo.esa.int ), the AGDC will allow analyses to be performed on a data store. By arranging EO data spatially and temporally, the AGDC enables efficient large-scale analysis using a ¿dice and stack¿ method which sub-divides the data into spatially regular, time-stamped, band -aggregated tiles that can be traversed as a dense temporal stack. The AGDC application programming interface (API) allows users to develop custom processing tasks. The API provides access to the tiles by abstracting the low level data access. Users don¿t need to be aware of the underlying system and data specific interactions to be able to formulate and execute processing tasks. The development of precision correction methodologies to enable production of comparable observations (spatially and spectrally), as well as the attribution of quality information about the contents of those observations is key to the success of the AGDC. Quality information for each observation is represented as a series of bitwise tests which, in the case of Landsat, include: contiguity of observations between layers in the dataset¿ cloud and cloud shadow obscuration¿ and a land/sea mask. Work in currently underway to further develop the open source solution from the initial prototype deployment. Components of the evolution include advancing the system design and function to provide: improved support for additional sensors¿ improved ingestion support¿ configurable storage units¿ provide high performance data structures¿ graphic user interface implementation and expanded collaboration and engagement. This paper reviews the history of the data cube and the application areas that will be addressed by the current plan of works. This research was undertaken with the assistance of resources from the National Computational Infrastructure (NCI), which is supported by the Australian Government. Presented at the 2016 Living Planet Symposium (LPS16) Prague, Czech Republic