The New world of 'Big Data' Analytics and High Performance Data: A Paradigm shift in the way we interact with very large Earth Observation datasets Purss B J Matthew, Lewis Adam, Ip Alexander, Evans Ben ABSTRACT The next decade promises an exponential increase in volumes of open data from Earth observing satellites. The ESA Sentinels, the Japan Meteorological Agency's Himawari 8/9 geostationary satellites, various NASA missions, and of course the many EO satellites planned from China, will produce petabyte scale datasets of national and global significance. It is vital that we develop new ways of managing, accessing and using this 'big-data' from satellites, to produce value added information within realistic timeframes. A paradigm shift is required away from traditional 'scene based (and labour intensive) approaches with data storage and delivery for processing at local sites, to emerging High Performance Data (HPD) models where the data are organised and co-located with High Performance Computational (HPC) infrastructures in a way that enables users to bring themselves, their algorithms and the HPC processing power to the data. Automated workflows, that allow the entire archive of data to be rapidly reprocessed from raw data to fully calibrated products, are a crucial requirement for the effective stewardship of these datasets. New concepts such as arranging and viewing data as 'data objects' which underpin the delivery of 'information as a service' are also integral to realising the transition into HPD analytics. As Australia's national remote sensing and geoscience agency, Geoscience Australia faces a pressing need to solve the problems of 'big-data', in particular around the 25-year archive of calibrated Landsat data. The challenge is to ensure standardised information can be extracted from the entire archive and applied to nationally significant problems in hazards, water management, land management, resource development and the environment. Ultimately, these uses justify government investment in these unique systems. A key challenge was how best to organise the archive of calibrated Landsat data (estimated to grow to almost 1 PB by the end of 2014) in a way that supports HPD applications yet with the ability to trace each observation (pixel) back to its original satellite acquisition. The approach taken was to develop a multi-dimensional array (a data cube) underpinned by the partitioning the data into tiles, without any temporal aggregation. This allows for flexible spatio-temporal queries of the archive whilst minimising the need to perform geospatial processing just to locate the pixels of interest. Equally important is the development and implementation of international data interoperability standards (such as OGC web services and ISO metadata standards) that will provide advanced access for users to interact with and query the data cube without needing to download any data or to go through specialised data portals. This new approach will vastly improve access to, and the impact of, Australia's Landsat archive holdings.

The dataset was created by Geoscience Australia using the framework described in Section 33 of the Offshore Petroleum and Greenhouse Gas Storage Act 2006 (OPGGS Act) and previous Acts. The international, scheduled areas and coastal waters used in this dataset are those found in the historical Australian Maritime Boundary Dataset 2006 (AMB2006). This version contains a number of technical amendments made on the basis of user feedback. These include: - Harmonisation of maritime boundaries, scheduled area and graticular block datasets in block SC51 The dataset is comprised of both polygons and points created to very high precision, accurate to within millimetres. The blocks have been cut by Australia's international boundaries, the scheduled areas and the coastal waters. Each block is assigned a polygon, including partial blocks. All blocks are titled with their block ID, and a list of vertices that make up the blocks. Each vertex of the dataset is also replicated as a discrete point in the points dataset. The design of the dataset allows for the exact location of every vertex to be known to millimetre precision. The corner coordinates of blocks are now defined to a high precision, and can be found by querying the appropriate point. Existing production licences AC/P17, AC/RL4, and AC/RL5 are included in the dataset. The blocks are attributed with fields containing information on: - Block ID - Parent 1 Million Mapsheet - Offshore Area - Epoch of the boundaries used to cut the data AMB2006 - Datum - Origin of the mapsheet in AGD66 - The position of all vertices in the block - The number of vertices in the block - The area of the block in acres - The area of the block in hectares The calculation used to find the area of the blocks is estimated to be precise to better than 1%. This is considered to be sufficient as under the permit and licensing arrangements in the OPGGS Act, the area of a block has no relevance. Therefore the area figure is provided solely for reference.

PLEASE NOTE: These data have been updated. See Related Links for new data. Geodatabase of the Commonwealth Seas and Submerged Lands Act 1973 - An Act relating to Sovereignty in respect of certain Waters of the Sea and in respect of the Airspace over, and the Sea bed and Subsoil beneath, those Waters and to Sovereign Rights in respect of the Continental Shelf and the Exclusive Economic Zone and to certain rights of control in respect of the Contiguous Zone.

PLEASE NOTE: These data have been updated. See Related Links for new data. Geodatabase of the Commonwealth Coastal Waters (State/Territory Powers) Act 1980 - An Act to extend the legislative powers of the States/Northern Territory in and in relation to coastal waters.

Australian Petroleum Blocks (2006) - Aligned with the historical Australian Maritime Boundary Dataset (AMB2006) V2.0 The digital realisation of Australia's Graticular Framework as set out under the Offshore Petroleum and Greenhouse Gas Storage Act 2006. Web Service Capabilities WMS,WFS,WCS. The dataset was created internally by Geoscience Australia using the framework described in Section 33 of the Offshore Petroleum and Greenhouse Gas Storage Act 2006 (OPGGS Act) and previous Acts. The international, scheduled areas and coastal waters used in this dataset are those found in the current Australian Maritime Boundary Dataset (AMB2006). The dataset is comprised of both polygons and points created to very high precision, accurate to within millimetres. The blocks have been cut by Australia's international boundaries, the scheduled areas and the coastal waters. Each block is assigned a polygon, including partial blocks. All blocks are titled with their block ID, and a list of vertices that make up the blocks. Each vertex of the dataset is also replicated as a discrete point in the points dataset. The design of the dataset allows for the exact location of every vertex to be know to millimetre precision. The corner coordinates of blocks are now defined to a high precision, and can be found by querying the appropriate point. Existing production licences AC/P17, AC/RL4, and AC/RL5 are included in the dataset. The blocks are attributed with fields containing information on: Block ID Parent 1 Million Mapsheet Offshore Area Epoch of the boundaries used to cut the data - AMB2006 Datum Origin of the mapsheet in AGD66 The position of all vertices in the block The number of vertices in the block The area of the block in acres The area of the block in hectares The calculation used to find the area of the blocks is estimated to be precise to better than 1%. This is considered to be sufficient as under the permit and licensing arrangements in the OPGGS Act, the area of a block has no relevance. Therefore the area figure is provided solely for reference.

Geoscience Australia has the primary role in the delineation of Australia's domestic and international maritime limits and boundaries. An output of this activity is the development of the Australian Maritime Boundaries (AMB) data. AMB2014 is a GIS data product, replacing AMBIS 2006, providing access to the data for Australia's maritime zones. Offshore Petroleum and Greenhouse Storage Act 2006 - Australian Maritime Boundaries Dataset 2014 is a digital representation of the scheduled and offshore areas defined under this Act. In the event of an inconsistency between AMB data and the limits under any legislation, the latter prevails. The data have been published by Geoscience Australia in consultation with other relevant Commonwealth Government agencies including the Attorney-General's Department, the Department of Foreign Affairs and Trade and the Australian Hydrographic Office. This version contains a number of technical amendments made on the basis of user feedback. These include: - Harmonisation of maritime boundaries, scheduled area and graticular block datasets in block SC51 Offshore Petroleum and Greenhouse Storage Act 2006 - Australian Maritime Boundaries Dataset 2014 comprise the scheduled and offshore areas defined under the Act. AMB Data can also be viewed on-line via AMSIS (http://www.ga.gov.au/marine/jurisdiction/amsis.html). Product Specifications Coverage: Australia including all external territories Currency: May 2014. Coordinates: Geographical Datum: GDA94 (functionally equivalent to WGS84). Format: ESRI Geodatabase and REST web service - free online. The area of coverage of the AMB data is for the whole of the Australian marine jurisdiction which includes waters adjacent to the mainland, offshore islands and External Territories. The geographical extent is approximately between the latitude limits of 8 S to 70 S and the longitude limits 39 E to 174 E. This dataset includes the following disclaimers: AMB data is a digital representation of the territorial sea baseline and of the outer limits of Australia's maritime zones. The baseline and zones are established under the Offshore Petroleum and Greenhouse Storage Act 2006. In the event of an inconsistency between AMB data and the areas under the legislation, the latter prevails.

The dataset was created by Geoscience Australia using the framework described in Section 33 of the Offshore Petroleum and Greenhouse Gas Storage Act 2006 (OPGGS Act) and previous Acts. The international, scheduled areas and coastal waters used in this dataset are those found in the historical Australian Maritime Boundary Dataset 2006 (AMB2006). This version contains a number of technical amendments made on the basis of user feedback. These include: - Harmonisation of maritime boundaries, scheduled area and graticular block datasets in block SC51 The dataset is comprised of both polygons and points created to very high precision, accurate to within millimetres. The blocks have been cut by Australia's international boundaries, the scheduled areas and the coastal waters. Each block is assigned a polygon, including partial blocks. All blocks are titled with their block ID, and a list of vertices that make up the blocks. Each vertex of the dataset is also replicated as a discrete point in the points dataset. The design of the dataset allows for the exact location of every vertex to be known to millimetre precision. The corner coordinates of blocks are now defined to a high precision, and can be found by querying the appropriate point. Existing production licences AC/P17, AC/RL4, and AC/RL5 are included in the dataset. The blocks are attributed with fields containing information on: - Block ID - Parent 1 Million Mapsheet - Offshore Area - Epoch of the boundaries used to cut the data AMB2001 - Datum - Origin of the mapsheet in AGD66 - The position of all vertices in the block - The number of vertices in the block - The area of the block in acres - The area of the block in hectares The calculation used to find the area of the blocks is estimated to be precise to better than 1%. This is considered to be sufficient as under the permit and licensing arrangements in the OPGGS Act, the area of a block has no relevance. Therefore the area figure is provided solely for reference.

Coping with the deluge of 'Big Data': The challenge of exploiting satellite earth observation data in the new era of High Performance Data Matthew B J Purss, Adam Lewis, Alexander Ip, Lesley Wyborn ABSTRACT Australia's Earth Observation Program has acquired and archived satellite data for the Australian Government since the establishment of the Australian Landsat Station in 1979. Data have been acquired from many sensors and platforms including ERS, EnviSAT, MODIS, ASTER, SPOT and ALOS, although the bulk of the continuous observations are from the Landsat instruments. The Landsat mission is the longest continuous environmental monitoring experiment in history; producing a global archive of earth observations spanning over 41 years. Geoscience Australia maintains an archive of Landsat data for Australia and produces products and information to support the delivery of government policy objectives. Future Earth observation missions promise an exponential increase in the volumes of open data from Earth observing satellites. For the Australian region the NASA/USGS Landsat-8 satellite is now contributing up to 50 GB of data per day and ESA's Sentinel-2 constellation (due for launch in early 2014) will provide close to 500 GB of data per day to Australia's existing archive of earth observation data. With just these two new data sources the Australian Satellite Earth Observation archive is expected to grow to around 1 PB by the end of 2014. Extracting information from satellite data is a long-standing challenge made more difficult by increased data volumes. Recognising this issue, the Australian Government funded the 'Unlocking the Landsat Archive' (ULA) consortium project from 2010 to 2013 to process Australia's Landsat archive to fully calibrated sensor and scene independent data products for the period from 1998 to 2012 and to investigate methods of arranging this archive so that it can be exploited to produce value added information products. The data outputs from the ULA project, currently totalling close to 400 TB, have become a fundamental component of Australia's eResearch infrastructure. The data are hosted on the National Computational Infrastructure (NCI) and are openly available under a Creative Commons licence. A key challenge for data custodians in this era of High Performance Data (HPD) is how to store and organise very large datasets that extend from terra-scale to peta-scale in a way that will facilitate data interoperability. Building on the foundation laid by the ULA project, Geoscience Australia is developing advanced data cube technologies that will enable the Australian Government to cope with the anticipated flow of new Earth observation data from future platforms. By collocating the data with the high performance computing capability of the NCI, and taking a standards approach to provide access to this data as a service, there is an opportunity to truly unlock the potential of Earth observation data to address questions in ways that previously were not possible.

Web services of the Commonwealth Seas and Submerged Lands Act 1973 - An Act relating to Sovereignty in respect of certain Waters of the Sea and in respect of the Airspace over, and the Sea bed and Subsoil beneath, those Waters and to Sovereign Rights in respect of the Continental Shelf and the Exclusive Economic Zone and to certain rights of control in respect of the Contiguous Zone.

The dataset was created by Geoscience Australia using the framework described in Section 33 of the Offshore Petroleum and Greenhouse Gas Storage Act 2006 (OPGGS Act) and previous Acts. The international, scheduled areas and coastal waters used in this dataset are those found in the current Australian Maritime Boundary Dataset 2014 (AMB2014a). This version contains a number of technical amendments made on the basis of user feedback. These include: - Removal of lease AC/P17 - Harmonisation of maritime boundaries, scheduled area and graticular block datasets in block SC51 The dataset is comprised of both polygons and points created to very high precision, accurate to within millimetres. The blocks have been cut by Australia's international boundaries, the scheduled areas and the coastal waters. Each block is assigned a polygon, including partial blocks. All blocks are titled with their block ID, and a list of vertices that make up the blocks. Each vertex of the dataset is also replicated as a discrete point in the points dataset. The design of the dataset allows for the exact location of every vertex to be known to millimetre precision. The corner coordinates of blocks are now defined to a high precision, and can be found by querying the appropriate point. Existing production licences AC/RL4, and AC/RL5 are included in the dataset. The blocks are attributed with fields containing information on: - Block ID - Parent 1 Million Mapsheet - Offshore Area - Epoch of the boundaries used to cut the data AMB2014 - Datum - Origin of the mapsheet in AGD66 - The position of all vertices in the block - The number of vertices in the block - The area of the block in acres - The area of the block in hectares The calculation used to find the area of the blocks is estimated to be precise to better than 1%. This is considered to be sufficient as under the permit and licensing arrangements in the OPGGS Act, the area of a block has no relevance. Therefore the area figure is provided solely for reference.