EMD-API: EMD-WRF On-Demand

Introduction

Mesoscale Modelling in 21 Lines of Code.

The EMD-API: EMD-WRF On-Demand is a part of EMD-API. It provides a unified interface to the EMD-WRF mesoscale on-demand services (see here and here). The service is aimed at detailed time-series analysis. EMD-API helps consultants, analysts and scientists working with high-resolution climate data in achieving their goals in an efficient way. EMD-API: EMD-WRF On-Demand has the following key-features:

• Global data delivery: All models available has a global footprint - so calculations are available at any location.
• On-demand calculation: Modelling is done at the high-performance computing (HPC) facilities at EMD.
• Multiple model configurations: We provide access to multiple model configurations, aimed at single points, larger areas and ice-loss calculations.
• Unified interface: EMD-API is a unified interface which allows for easy integration to internal processes and tools.
• Trusted datasets: EMD-API builds upon the trusted models, data-bases and data-sources that have been used through the online-data services in windPRO for more than a decade.
• Built on open standards: EMD-API is a REST based service that implements the OpenAPI standard. Results are in the open .mesores data-format.
• Available from any development tool: Service access is available from your preferred development platform - like C#, R, python, html, java, php, scala and swift (just use the OpenAPI tools to generate the client software for your preferred platform).

Access and Requirements

To see more documentation and to access the calculation-services, please visit the API-documentation through the following URL's:

• EMD-API Overview (Wiki) - here.
• EMD-API Main Page (API) - here.
• EMD-API Climate Data UI (API) - here.

Any technical questions on our calculation service can be addressed to our Senior Technical Specialist - Morten Lybech Thøgersen: mlt@emd.dk.

Requirements:
In order to use the EMD-API: EMD-WRF On-Demand, you will need to:

1. Have an API access token for your user and company account. Note: Your access token is personal and will allow to operate the EMD-API: EMD-WRF On-Demand. You can also browse and view credit-status and pending calculations for your company in read-only model.
2. Have some calculation credits on your account (a calculation credit corresponds to one month of mesoscale time-series data in the standard 1-point model configuration)

Calculation results are delivered in the .mesores open-data format for easy integration in windPRO and other tools such as python (pandas) and R. Read more here.

Available Models and Cost

The following mesoscale model configurations are currently available through the EMD-API. The cost per month is 1 calculation credit (CPU=Cost-Per-Unit) for the 1-point calculations, 2 CPU for the 9-point and 3 CPU for the 25 point calculation. The icing configuration is 2-CPU per month of time-series. Since the calculations are run on-demand by the aid of 1000's of computer-cores, the calculation will last a while (several hours to days) to complete. The table below shows the models available and data-status as of 2023-05-24; an updated model-table can be obtained directly from the API.

EMD-API AVAILABLE MODELS
------------------------
description               model_id                          start_date            end_date              supported_areas
------------------------  --------------------------------  --------------------  --------------------  ------------------------------------
ERA5 3km GlobCover        emd_36_3km_era5hourly_glob        1999-01-01T00:00:00Z  2023-05-01T00:00:00Z  ['1 point', '9 points', '25 points']
ERA5 3km Icing            emd_36_3km_era5hourly_icing_glob  1999-01-01T00:00:00Z  2023-05-01T00:00:00Z  ['1 point']
CFSR/CFSv2 3km GlobCover  emd_36_3km_cfsr_glob              1994-01-01T00:00:00Z  2023-05-01T00:00:00Z  ['1 point', '9 points', '25 points']

Data Model - EMD-API: EMD-WRF On-Demand

The EMD-API: EMD-WRF On-Demand service is documented in a REST based API using the OpenAPI Specification. You can view the interfaces and download the interfaces as json or yaml, see more here. The service provides the following functionality:

• List Models: Full list of available EMD-WRF On-Demand model configurations including their ID’s and other meta-data.
• Information on used and purchased credits: Request information of completed, cancelled and pending calculations for your company account.
• Place Order: Order a mesoscale time-series data for any location with any model (dataset) - given any latitude-longitude location. You can decide which period to generate.
• Order Status: Request progress for a specific order - and eventually recieve the download URL for the order. Options: [PENDING, CANCELLED OR SUCCESS].
• Cancel Order: Cancel an order that has not started yet (status must be PENDING).

Python - Installation and Example Code

These examples provides a demonstration on how to use the EMD-API: EMD-WRF On-Demand with python installed in a virtual environment. If you are using CONDA or MINICONDA, we recommend that you create a new virtual environment and use a recent 3.x version of python. When the virtual environment is created, then activate the environment and install the required packages.

Open your Anaconda Prompt. Copy-paste the following lines, one-by-one:

conda create -n emdapiclient_emdwrfod python=3.8
conda activate emdapiclient_emdwrfod
conda install -c conda-forge pandas requests tabulate

Python - Example Code

In order to test your setup and learn to use the EMD-API: EMD-WRF On-Demand, we suggest that you download and run the three examples that we have created - here.
Unpack the zip files, activate your python environment and run the command below in your integrated-environment, terminal or command-shell.

Then work your way through through each example provided. Each of the three python samples holds a specific focus:

1. emdwrfod_example_status.py: Status and overview of user-request and past-calculations.
2. emdwrfod_example_order_submit.py: Submit one or more calculations.
3. emdwrfod_example_order_check.py: Check calculation status and retrieve data from one or more calculations.

Python - Minimalistic Example - 20 lines

The global-coverage EMD-WRF On-Demand service is making it super-easy to integrate advanced mesoscale modelling into custom and automated software-driven workflows. These few lines of code demonstrate how you can set up an EMD-WRF on-demand mesoscale model with selected boundary data, spinning-up 1000’s of CPU cores and finally – after some hours of calculation time - to obtain one or more hourly times-series data-nodes in an open format - with more 100 climate variables ready for further analysis in your favorite software.

import requests, json, time, os, shutil

api_url = "https://api.emd.dk/emd-wrf-od/"
request_data = {"lat": 57.50, "lon": 10.50, "start_date": "2013-01-01", "end_date": "2022-12-31", "model_id": 'emd_36_3km_era5hourly_glob', "area": "9 points"}
api_key = 'ADD_YOUR_OWN_API_KEY_HERE'   

response_submit = requests.post(url = api_url + "order_points", params=request_data, headers={'X-Auth': api_key}, verify=True)
order = json.loads(response_submit.content)
while True:
    response_status = requests.get(api_url + "status/" + order['order_id'], headers={'X-Auth': api_key}, verify=True)
    order_status = json.loads(response_status.content)
    if order_status["download_url"]:
        download_response = requests.get(order_status["download_url"], stream=True)
        with open(os.path.join("c:/temp/", os.path.basename(order_status["download_url"]), 'wb')) as mesoresfile:
            shutil.copyfileobj(download_response.raw, mesoresfile)
        break
    else:
        print(time.strftime("%H:%M:%S", time.localtime()), " - processing - ", order_status["status"], order_status["estimated_completion_time"], order_status["message"])
        time.sleep(60)
print(time.strftime("%H:%M:%S", time.localtime()), " - Done: Mesoscale Modelling. Total of 20 lines of python code.")

Client Software Generation: Many Languages and Tools

REST and OpenAPI is easily consumed from a lot of software tools. It is perfectly possible that your preferred language is supported. OpenAPI works well with languages such as - but not limited to - C#, R, python, java, php, scala and swift. Just download the YAML or JSON service description and use the Swagger Editor or OpenAPI Generator to generate the client libraries for your preferred software. Then you are ready to integrate towards your preferred systems and workflows.

To generate the client libries yourself - one possible process is to:

1. Download the OpenAPI (openapi.yaml or openapi.json) description files - here-yaml or here-json
2. Load it into the swagger editor - here
3. Choose to "Generate Client" from the drop-down menu within the swagger editor.