Web Service to Order ARM Data

Table of Contents


Getting Started

To get started with the ADRS Web Service please do the following:


Restrictions

This web service is designed for limited use by ARM users who need to order raw files or specific versions of files, other than the latest version.

Overview of the web service

Registered users of ARM data, who have been provided with access keys, may use this web service to order ARM data outside of the ARM Data Discovery Tool.

The web service provides the following functionality:

The web service accepts a JSON object, via the POST method. The JSON object contains the various parameters that the web service needs to service a request.

Details of web service:

URL: https://archive.arm.gov/cgi-bin/adrsws.py

Number of request parameters: 1

Request parameter name: reqjson

Request JSON object specification:

When reqtype == "dslist"
Key Name Requirement Description
accesskey Mandatory A unique key given to a user of this web service. Keys can be shared within a group. The purpose of this key to prevent DoS (Denial of Service) attacks and unauthorized placement of orders for ARM data.
userid Mandatory The user ID of a registered ARM user in good standing.
reqtype Mandatory Value should be: "dslist"


When "reqtype": "dates"
Key Name Requirement Description
accesskey Mandatory See above
userid Mandatory See above
reqtype Mandatory Value should be: "dates"
datastream Mandatory Full data stream name. (e.g., sgp30baebbrE39.c1)


When "reqtype": "flist"
Key Name Requirement Description
accesskey Mandatory See above
userid Mandatory See above
reqtype Mandatory Value should be: "flist"
withvers Optional Takes either "y" or "n". If "y", then the version extensions on the retrieved filenames are retained.
datastream Mandatory Full data stream name. (e.g., sgp30baebbrE39.c1)
startdate Mandatory Start date in the following format: YYYY-mm-dd
enddate Mandatory End date in the following format: YYYY-mm-dd


When reqtype == "order"
Key Name Requirement Description
accesskey Mandatory See above
userid Mandatory See above
reqtype Mandatory Value should be: "order"
withvers Optional Takes either "y" or "n". If "y", then the version extensions on the retrieved files are retained.
fileslist Mandatory A list containing the file names to be retrieved. If version extension is specified, then that exact version is retrieved. If no version extension is supplied, the latest version is returned. Duplicates in the list are automatically removed and invalid file names are ignored.


When reqtype == "userid"
Key Name Requirement Description
accesskey Mandatory See above
reqtype Mandatory Value should be: "userid"
email Mandatory Email address of the user as defined in the ARM database


Examples:

Request JSON where user "<your userid>" requests list of all datastreams:

{
  "accesskey": "__ACCESS_KEY_FROM_ARM_ADC__",
  "userid": "<your userid>",
  "reqtype": "dslist"
}

Request JSON where user "<your userid>" requests list of files from sgpmfrsraod1michE31.c1 datastream for a date range from 2016-01-01 to 2016-02-01, with version designators left intact on the filenames:

{
  "accesskey": "__ACCESS_KEY_FROM_ARM_ADC__",
  "userid": "<your userid>",
  "withvers": "n",
  "reqtype": "flist",
  "datastream": "sgpmfrsraod1michE31.c1",
  "startdate": "2016-01-01",
  "enddate": "2016-02-01"
}

Request JSON where user "<your userid>" orders 3 files:

{
  "accesskey": "__ACCESS_KEY_FROM_ARM_ADC__",
  "userid": "<your userid>",
  "withvers": "n",
  "reqtype": "order",
  "fileslist": [
    "sgpmfrsraod1michE31.c1.20160101.000000.cdf",
    "sgpmfrsraod1michE31.c1.20160102.000000.cdf",
    "sgpmfrsraod1michE31.c1.20160103.000000.cdf"
  ]
}

Convenience shell script

A convenience shell script that will create the required JSON object and make the call to the web service is provided. It can be downloaded from:
https://adc.arm.gov/docs/adrsws-sh.tgz     [MD5(adrsws-sh.tgz)= 28502620442c3b10d5fdb60940130829]

Usage examples

To find out the ARM User ID of a user, given the user's email address: 
  adrsws.sh -t userid -a <email address>
To get a list of all datastreams as the above sample user, <your userid>: 
  adrsws.sh -t dslist -u <your userid>
To get the start and end dates for a datastream: 
  adrsws.sh -t dates -u <your userid> -d
To get a list of files from the sgpmfrsraod1michE31.c1 datastream for a date range from 2016-01-01 to 2016-02-01, with version designators left intact on the filenames, again for our sample user: 
  adrsws.sh -u <your userid> -t flist -v -d sgpmfrsraod1michE31.c1 -s 2016-01-01 -e 2016-02-01
To collect those file names in plain text format (one file name per line), instead of the default JSON format, in a file called "filelist.txt", use the "-p" flag. The command would then be: 
  adrsws.sh -u <your userid> -t flist -v -d sgpmfrsraod1michE31.c1 -s 2016-01-01 -e 2016-02-01 -p > filelist.txt
To order specific files defined in "filelist.txt", with version designators intact (ex. sgpmfrsraod1michE31.c1.20150101.000000.cdf.v1), the command would be: 
  adrsws.sh -u <your userid> -v filelist.txt
or 
  cat filelist.txt | adrsws.sh -u <your userid> -v

Pipelining is also supported. For example, to get a list of file names and place the order in one consolidated step, the command would be: 
  adrsws.sh -u <your userid> -t flist -d sgpmfrsraod1michE31.c1 -s 2016-01-01 -e 2016-02-01 -p \
  | adrsws.sh -u <your userid>

To have the ordered files automatically downloaded to a specific destination directory, say /home/<your userid>/arm-downloads, the command would be: 
  adrsws.sh -u <your userid> -t flist -d sgpmfrsraod1michE31.c1 -s 2016-01-01 -e 2016-02-01 -p \
  | adrsws.sh -u <your userid> -g /home/<your userid>/arm-downloads

Running: 
  adrsws.sh
without any arguments, will display detailed help on usage.

OUTPUT INFORMATION

The output from the web service and convenience shell script will be JSON. The convenience shell script can take a "-p" flag to have the output in plain text. The "-p" flag only works when combined with "-t dslist" or "-t flist" flags.

Help/Questions

To request an access key or for help/questions, please send email to:
armarchive@ornl.gov