Authors / CoAuthors
Purss, M. | Lewis, A. | Ip, A. | Evans, B.
Abstract
The New world of 'Big Data' Analytics and High Performance Data: A Paradigm shift in the way we interact with very large Earth Observation datasets Purss B J Matthew, Lewis Adam, Ip Alexander, Evans Ben ABSTRACT The next decade promises an exponential increase in volumes of open data from Earth observing satellites. The ESA Sentinels, the Japan Meteorological Agency's Himawari 8/9 geostationary satellites, various NASA missions, and of course the many EO satellites planned from China, will produce petabyte scale datasets of national and global significance. It is vital that we develop new ways of managing, accessing and using this 'big-data' from satellites, to produce value added information within realistic timeframes. A paradigm shift is required away from traditional 'scene based (and labour intensive) approaches with data storage and delivery for processing at local sites, to emerging High Performance Data (HPD) models where the data are organised and co-located with High Performance Computational (HPC) infrastructures in a way that enables users to bring themselves, their algorithms and the HPC processing power to the data. Automated workflows, that allow the entire archive of data to be rapidly reprocessed from raw data to fully calibrated products, are a crucial requirement for the effective stewardship of these datasets. New concepts such as arranging and viewing data as 'data objects' which underpin the delivery of 'information as a service' are also integral to realising the transition into HPD analytics. As Australia's national remote sensing and geoscience agency, Geoscience Australia faces a pressing need to solve the problems of 'big-data', in particular around the 25-year archive of calibrated Landsat data. The challenge is to ensure standardised information can be extracted from the entire archive and applied to nationally significant problems in hazards, water management, land management, resource development and the environment. Ultimately, these uses justify government investment in these unique systems. A key challenge was how best to organise the archive of calibrated Landsat data (estimated to grow to almost 1 PB by the end of 2014) in a way that supports HPD applications yet with the ability to trace each observation (pixel) back to its original satellite acquisition. The approach taken was to develop a multi-dimensional array (a data cube) underpinned by the partitioning the data into tiles, without any temporal aggregation. This allows for flexible spatio-temporal queries of the archive whilst minimising the need to perform geospatial processing just to locate the pixels of interest. Equally important is the development and implementation of international data interoperability standards (such as OGC web services and ISO metadata standards) that will provide advanced access for users to interact with and query the data cube without needing to download any data or to go through specialised data portals. This new approach will vastly improve access to, and the impact of, Australia's Landsat archive holdings.
Product Type
nonGeographicDataset
eCat Id
77439
Contact for the resource
Custodian
Owner
Custodian
Cnr Jerrabomberra Ave and Hindmarsh Dr GPO Box 378
Canberra
ACT
2601
Australia
Keywords
-
- External Publication
- Australian and New Zealand Standard Research Classification (ANZSRC)
-
- Geospatial Information Systems
-
- Published_Internal
Publication Date
2013-01-01T00:00:00
Creation Date
Security Constraints
Legal Constraints
Status
Purpose
Maintenance Information
asNeeded
Topic Category
geoscientificInformation
Series Information
Lineage
abstract
Parent Information
Extents
Reference System
Spatial Resolution
Service Information
Associations
Downloads and Links
Source Information
Source data not available.