Skip to content

Commit

Permalink
Merge branch 'release/1.5.1'
Browse files Browse the repository at this point in the history
  • Loading branch information
Cesare Rossi committed Jan 21, 2016
2 parents d930265 + af8c255 commit 3588dc1
Show file tree
Hide file tree
Showing 3 changed files with 78 additions and 39 deletions.
82 changes: 54 additions & 28 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,70 +1,96 @@
Cloud processing with Envisat ASAR data and ROI_PAC
=
## Developer Cloud Sandbox interferogram processing with ROI_PAC

<a href="http://dx.doi.org/10.5281/zenodo.10015"><img src="https://zenodo.org/badge/doi/10.5281/zenodo.10015.png"></a>

This repository contains the application files and scripts to process a pair (Master and Slave) of Envisat ASAR data with [ROI_PAC](http://www.openchannelfoundation.org/projects/ROI_PAC) (Repeat Orbit Interferometry PACkage), a software package jointly created by the Jet Propulsion Laboratory division of NASA and CalTech for processing SAR data to create InSAR (Interferometric synthetic aperture radar) images, or 'interferograms'. This geodetic method uses two or more synthetic aperture radar (SAR) scenes to generate maps of surface deformation or digital elevation models, using differences in the phase of the waves returning to the radar sensor. The technique can potentially measure centimetre-scale changes in deformation over spans of days to years. It has applications for geophysical monitoring of natural hazards, for example earthquakes, volcanoes and landslides, and in structural engineering, in particular monitoring of subsidence and structural stability.

### Getting Started
## Quick link

To run this application you will need a Developer Cloud Sandbox, that can be either requested from the ESA [Research & Service Support Portal](http://eogrid.esrin.esa.int/cloudtoolbox/) for ESA G-POD related projects and ESA registered user accounts, or directly from [Terradue's Portal](http://www.terradue.com/partners), provided user registration approval.
* [Getting Started](#getting-started)
* [Installation](#installation)
* [Submitting the workflow](#submit)
* [Community and Documentation](#community)
* [Authors](#authors)
* [Questions, bugs, and suggestions](#questions)
* [License](#license)

### <a name="getting-started"></a>Getting Started

To run this application you will need a Developer Cloud Sandbox, that can be either requested from:
* ESA [Geohazards Exploitation Platform](https://geohazards-tep.eo.esa.int) for GEP early adopters;
* ESA [Research & Service Support Portal](http://eogrid.esrin.esa.int/cloudtoolbox/) for ESA G-POD related projects and ESA registered user accounts
* From [Terradue's Portal](http://www.terradue.com/partners), provided user registration approval.

A Developer Cloud Sandbox provides Earth Sciences data access services, and helper tools for a user to implement, test and validate a scalable data processing application. It offers a dedicated virtual machine and a Cloud Computing environment.
The virtual machine runs in two different lifecycle modes: Sandbox mode and Cluster mode.
Used in Sandbox mode (single virtual machine), it supports cluster simulation and provides user assistance functions in building the distributed application.
Used in Cluster mode (a set of master and slaves nodes), it supports the deployment and execution of the application with the power of distributed computing for data processing over large datasets (leveraging the Hadoop Streaming MapReduce technology).
The virtual machine runs in two different lifecycle modes: Sandbox mode and Cluster mode.
Used in Sandbox mode (single virtual machine), it supports cluster simulation and user assistance functions in building the distributed application.
Used in Cluster mode (a set of master and slave nodes), it supports the deployment and execution of the application with the power of distributed computing for data processing over large datasets (leveraging the Hadoop Streaming MapReduce technology).

### Installation
### <a name="installation"></a>Installation

Log on your Developer Cloud Sandbox host.
#### Pre-requisites

Install ROI_PAC using the 'yum' command.
ROI-PAC is a software package available through the Terradue Cloud Platform software repository.

> ROI_PAC is a copyrighted software that requires a license. Licenses are available at no charge for non-commercial use from the [Open Channel Foundation](http://www.openchannelfoundation.org/projects/ROI_PAC). Read the [license terms](http://www.openchannelfoundation.org/project/print_license.php?group_id=282&license_id=61).
```
sudo yum install roi_pac.x86_64 roi_pac-grdfile.x86_64
sudo yum install roi_pac.x86_64 roi_pac-grdfile.x86_64
```

##### Using the releases

Log on the developer cloud sandbox. Download the RPM package from https://github.com/geohazards-tep/InSAR-ROI_PAC/releases.
Install the downloaded package by running these commands in a shell:

```bash
sudo yum -y install InSAR-ROI_PAC-<version>-ciop.x86_64.rpm
```

Run these commands in a shell:
#### Using the development version

Log on the developer sandbox and run these commands in a shell:

```bash
cd
git clone git@github.com:Terradue/InSAR-ROI_PAC.git
git clone https://github.com/Terradue/InSAR-ROI_PAC.git
cd InSAR-ROI_PAC
git checkout develop
mvn install
```

### Submitting the processing request
### <a name="submit"></a>Submitting the workflow

Invoke the ROI_PAC processing via the Cloud Sandbox's embedded OGC Web Processing Service (WPS) interface, using either:

* the WPS client in your Cloud Sandbox's [dashboard](http://docs.terradue.com/developer-sandbox/start/discover.html#user-dashboard-web-ui)
* your Browser, with a GET request to the WPS access point
* a POST request to the WPS access point

Learn more: [Submit a processing request via the OGC WPS interface](http://docs.terradue.com/developer-sandbox/faq)
Run this command in a shell:

```bash
ciop-run
```
Or invoke the Web Processing Service via the Sandbox dashboard or the [Geohazards Thematic Exploitation platform](https://geohazards-tep.eo.esa.int) providing a master and a slave products' URL, e.g.:

### Community and Documentation
* http://catalogue.terradue.int/catalogue/search/ASA_IM__0P/ASA_IM__0CNPDE20090412_092426_000000162078_00079_37207_1556.N1/rdf (master)
* http://catalogue.terradue.int/catalogue/search/ASA_IM__0P/ASA_IM__0CNPAM20080427_092430_000000172068_00079_32197_3368.N1/rdf (slave)

To learn more and find information go to
To learn more and find information go to

* [Developer Cloud Sandboxes](http://docs.terradue.com/developer-sandbox) service documentation
* [Developer Cloud Sandbox](http://docs.terradue.com/developer) service
* [InSAR ROI_PAC](http://docs.terradue.com/developer/field/insar/tp_roi_pac) field guide chapter
* [ESA Geohazards Exploitation Platform](https://geohazards-tep.eo.esa.int)

### Authors of this tutorial
### <a name="authors"></a>Authors (alphabetically)

* Francesco Barchetta
* Fabrice Brito
* Fabio D'Andria
* Emmanuel Mathot
* Cesare Rossi

### License for this tutorial
### <a name="questions"></a>Questions, bugs, and suggestions

Copyright 2014 Terradue Srl
Please file any bugs or questions as [issues](https://github.com/geohazards-tep/InSAR-ROI_PAC/issues/new) or send in a pull request.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0
### <a name="license"></a>License

Copyright 2015 Terradue Srl

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0
2 changes: 1 addition & 1 deletion pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
<packaging>rpm</packaging>
<name>InSAR-ROI_PAC</name>
<description>InSAR-ROI_PAC</description>
<version>1.3-SNAPSHOT</version>
<version>1.5.1</version>

<scm>
<connection>scm:git:[email protected]:Terradue/InSAR-ROI_PAC.git</connection>
Expand Down
33 changes: 23 additions & 10 deletions src/main/app-resources/roipac/run.sh
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash

# source the ciop functions (e.g. ciop-log)
source ${ciop_job_include}

Expand Down Expand Up @@ -32,7 +32,7 @@ function cleanExit ()

trap cleanExit EXIT

# create a shorter TMPDIR name for some ROI_PAC scripts/binaires
# create a shorter TMPDIR name for some ROI_PAC scripts/binaires
UUIDTMP="/tmp/`uuidgen`"
#ln -s $TMPDIR $UUIDTMP
mkdir ${UUIDTMP}
Expand All @@ -53,7 +53,7 @@ cat > $TMPDIR/input

# retrieve the aux files
mkdir -p $TMPDIR/aux
for input in `cat $TMPDIR/input | grep 'aux='`
for input in `cat $TMPDIR/input | grep 'aux='`
do
ciop-log "DEBUG" "retrieving aux file ${input#aux=}"
echo ${input#aux=} | ciop-copy -O $TMPDIR/aux - 2> /dev/null
Expand All @@ -62,7 +62,7 @@ done

# retrieve the orbit data
mkdir -p $TMPDIR/vor
for input in `cat $TMPDIR/input | grep 'vor='`
for input in `cat $TMPDIR/input | grep 'vor='`
do
ciop-log "DEBUG" "retrieving orbit file ${input#aux=}"
echo ${input#vor=} | ciop-copy -O $TMPDIR/vor - 2> /dev/null
Expand All @@ -78,7 +78,7 @@ ciop-log "DEBUG" "dem wps results is ${wps_result}"

# extract the result URL
curl -L -o $TMPDIR/workdir/dem/dem.tgz "${wps_result}" 2> /dev/null
tar xzf $TMPDIR/workdir/dem/dem.tgz -C $TMPDIR/workdir/dem/
tar xzf $TMPDIR/workdir/dem/dem.tgz -C $TMPDIR/workdir/dem/

dem="`find $TMPDIR/workdir/dem -name "*.dem"`"

Expand All @@ -94,7 +94,7 @@ do
# get the date in format YYMMDD
sar_date=`opensearch-client $sar_url startdate | cut -c 3-10 | tr -d "-"`
sar_date_short=`echo $sar_date | cut -c 1-4`

ciop-log "DEBUG" "SAR input ${sar_url}"
ciop-log "INFO" "SAR date: $sar_date and $sar_date_short"

Expand All @@ -104,7 +104,7 @@ do

sar_folder=$TMPDIR/workdir/$sar_date
mkdir -p $sar_folder

# get ASAR products
sar_url=`opensearch-client $sar_url enclosure`
sar="`ciop-copy -o $sar_folder $sar_url`"
Expand All @@ -126,7 +126,7 @@ do
base=${sar1}_${sar2}
geodir=${geodir}-${sar_date_short}
fi
done
done

ciop-log "INFO" "Conversion of SAR pair to RAW completed"

Expand Down Expand Up @@ -177,7 +177,10 @@ process_2pass.pl $roipac_proc 1>&2
cd int_${intdir}

ciop-log "INFO" "Geocoding the wrapped interferogram"
cpx2rmg filt_${intdir}-sim_HDR_4rlks.int filt_${intdir}-sim_HDR_4rlks.int.hgt
h=$(cat filt_${intdir}-sim_HDR_4rlks.int.rsc | grep FILE_LENGTH | tr -s " " | cut -d " " -f 2)
w=$(cat filt_${intdir}-sim_HDR_4rlks.int.rsc | grep WIDTH | tr -s " " | cut -d " " -f 2)
cpx2rmg filt_${intdir}-sim_HDR_4rlks.int filt_${intdir}-sim_HDR_4rlks.int.hgt ${w} ${h}
cp filt_${intdir}-sim_HDR_4rlks.int.rsc filt_${intdir}-sim_HDR_4rlks.int.hgt.rsc
geocode.pl geomap_4rlks.trans filt_${intdir}-sim_HDR_4rlks.int.hgt geo_${intdir}.int

ciop-log "INFO" "Creating geotif files for interferogram phase and magnitude"
Expand All @@ -188,6 +191,16 @@ gdal_translate NETCDF:"geo_${intdir}.int.nc":phase geo_${intdir}.int.phase.tif
gdal_translate NETCDF:"geo_${intdir}.int.nc":magnitude geo_${intdir}.int.magnitude.tif
gdal_translate NETCDF:"geo_${intdir}.unw.nc":phase geo_${intdir}.unw.phase.tif

# create quicklooks
# rescale
gdal_translate -scale -10 10 0 255 -ot Byte -of GTiff geo_${intdir}.unw.phase.tif geo_${intdir}.unw.phase.temp.tif
gdal_translate -scale -10 10 0 255 -ot Byte -of PNG geo_${intdir}.unw.phase.tif geo_${intdir}.unw.phase.png
listgeo -tfw geo_${intdir}.unw.phase.tif
mv geo_${intdir}.unw.phase.tfw geo_${intdir}.unw.phase.pngw

ciop-publish -m $TMPDIR/workdir/int_${intdir}/geo_${intdir}.unw.phase.png
ciop-publish -m $TMPDIR/workdir/int_${intdir}/geo_${intdir}.unw.phase.pngw

ciop-log "INFO" "Publishing results"
ciop-publish -m $TMPDIR/workdir/*.proc

Expand Down Expand Up @@ -226,6 +239,6 @@ ciop-publish -m $TMPDIR/workdir/int_${intdir}/$intdir.int
ciop-publish -m $TMPDIR/workdir/int_${intdir}/$intdir.int.rsc


rm -fr $UUIDTMP
#rm -fr $UUIDTMP

ciop-log "INFO" "That's all folks"

0 comments on commit 3588dc1

Please sign in to comment.