EMD-API - Climate Data Access

Introduction

The climate data access service of the EMD-API is a software library from EMD International: It delivers a unified interface to a wide range of climate data for detailed time-series analysis. EMD-API helps consultants, analysts and scientists working with high-resolution climate data in achieving their goals in an efficient way. It has the following key-features:

• Instant data delivery: All datasets within the EMDAPI are ready processed and requests are served within seconds or minutes
• 40+ climate datasets: EMDAPI provides access more than 40 of the best local, regional and global climate datasets and allows access to more than 1Pb of data.
• Unified interface: The unified interface which allows for integration to internal processes and tools - and also very efficient uncertainty analysis with gigabytes of data easily accessed.
• Trusted datasets: EMDAPI builds upon the trusted data-bases and data-sources that have been used through the online-data services in windPRO for more than a decade.
• Built on open standards: EMDAPI is a REST based service that implements the OpenAPI standard].
• Available from any development tool: Access to the climate databases is available from your preferred development platform - C#, R, python, html, java, php, scala and swift. Just use the OpenAPI tools to generate the client software for your preferred platform.

Access

To see more documentation and to access the data-services, please visit the API through the following URL's:

• EMD-API Overview (Wiki) - here.
• EMD-API Main Page (API) - here.
• EMD-API Climate Data UI (API) - here.

Any technical questions on our climate databases can be addressed to our Senior Technical Specialist - Morten Lybech Thøgersen: mlt@emd.dk.

Usage Constraints and Restrictions

Each API-access-token provides a full, virtually unrestricted access to the mesoscale and climate time-series datasets. So, even though data access is ‘flat rate’ and without any limitations on the number downloads or number of climate-variables – it is not permitted to do parallel downloads from the same API-token. This is to ensure enough bandwith and a good experience for all API users. This restriction is - from november 2022 - enforced through "rate limiting" - allowing up to 20 API-data-requests per running 10-minute interval. If you exceed this limit, a "HTTP ERROR 429: TOO MANY REQUESTS" is shown until below the limit again. Only the following endpoints are part of the rate-limiting procedure:

• /order/

Our recommended approach for API-processing and download is as from the pesudo-code below:

prepare list of nodes to download (from "coverage checks")
loop through list:
  request to download point
  wait at minuimum 30 seconds for api-server to finalize data-processing (until status = "success")
  download node from provided url

Data Model - Climate Data Service

The EMD climate-data API-services is documented in a REST based API using the OpenAPI Specification. You can view the interfaces and download the interfaces as json or yaml, here. The service provides the following functionality:

• List Datasets: Full list of available datasets with their ID’s, descriptions and documentation liks (URL's). This includes any private datasets connected to your user account.
• Locate Data-Nodes: Request locations of several nearby data nodes from a specific dataset given a latitude-longitude location.
• Update Dates: Information of when a dataset is updated
• Place Order: Time-series data for any location whith any dataset (from latitude-longitude location). You can decide which period to download.
• Order Status: Request progress for an order - and recieve the download URL for the order.

Python - Installation

Data nodes near the Egmond Aan Zee Offhore

The simplest way to use the EMDAPI with python is to install the client software in a virtual environment. If you are using CONDA or MINICONDA, we recommend that you create a new virtual environment and use a recent 3.x version of python. When the virtual environment is created, then activate the environment.

Open your Anaconda Prompt. Copy-paste the following lines:

conda create -n emdapiclient python=3.8.5
conda activate emdapiclient

Install the required packages needed in order to do data-science and use the examples provided within the jupyter notebooks. We have have validated this setup using specific package versions (used in the commands below).

In the Anaconda Prompt, copy-paste the following lines, one by one:

conda install -c conda-forge pandas=1.1.0 numpy=1.19.1
conda install -c conda-forge matplotlib=3.3.1 basemap=1.2.2 basemap-data-hires=1.2.2 
conda install -c conda-forge jupyter=1.0.0 ipykernel=5.3.4 
conda install -c conda-forge python-wget

Download the zipped-file holding the OpenAPI python client.
Unpack the file and install it within your virtual environment:

In the Anaconda Prompt: Move to the folder, where you have unpacked the zipped file. Copy-paste the following line:

python setup.py install

Make sure that the new emdapi virtual enviroment (python-kernel) is available to be used with jupyter-notebook environment:

python -m ipykernel install --user --name=emdapiclient

Python - Jupyter Notebooks for Demonstration and Test

In order to test your setup and learn to use the EMDAPI, we suggest that you download the jupyter-notebook examples that we have created - here.
Unpack the zip files and run the command below in your terminal or command-shell.
If jupyter prompts for you to select another python-kernel, then select the emdapiclient kernel (may also be selected directly from the 'Kernel' drop-down menu).

In the Anaconda Prompt: Move to the folder, where you have saved the jupyter-notebook examples. Copy paste the following line to open jupyter notebook from where you can open the examples.

jupyter notebook

Within the internet-browser (and jupyter user-interface), run select the notebook file (*.ipynb).
Then work your way through through each example provided. Each jupyter notebook holds a separate topic:

1. emdapi_availability.ipynb Demonstration of login to the system, then check which datasets are available at a specific location.
2. emdapi_correlation.ipynb: Correlation analysis (R2), check which climate datasets correlates the best against the Egmond Aan Zee data
3. emdapi_random.ipynb: Download climate data from any part of the world.

Client Software Other Languages and Tools

REST and OpenAPI is easily consumed from a lot of software tools. It is perfectly possible that your preferred language is supported. OpenAPI works well with languages such as - but not limited to - C#, R, python, java, php, scala and swift. Just download the YAML or JSON service description and use the Swagger Editor or OpenAPI Generator to generate the client libraries for your preferred software. Then you are ready to integrate towards your preferred systems and workflows.

To generate the client libries yourself - one possible process is to:

1. Download the OpenAPI (openapi.yaml or openapi.json) description files - here-yaml or here-json
2. Load it into the swagger editor - here
3. Choose to "Generate Client" from the drop-down menu within the swagger editor.