View source for EMD-API - Climate Data Access

[[Category:EMD-API]]
== Introduction ==
[[File:EMDAPI_451x303.jpg|thumb|400px|right]]The climate data access service of the EMD-API is a software library from EMD International:  It delivers a unified interface to a wide range of climate data for detailed time-series analysis. EMD-API helps consultants, analysts and scientists working with high-resolution climate data in achieving their goals in an efficient way. It has the following key-features: 

* '''Instant data delivery''': All datasets within the EMDAPI are ready processed and requests are served within seconds or minutes
* '''40+ climate datasets''': EMDAPI provides access more than 40 of the best local, regional and global climate datasets and allows access to more than 1Pb of data. 
* '''Unified interface''': The unified interface which allows for integration to internal processes and tools - and also very efficient uncertainty analysis with gigabytes of data easily accessed. 
* '''Trusted datasets''': EMDAPI builds upon the trusted data-bases and data-sources that have been used through the [http://help.emd.dk/mediawiki/index.php?title=Main_Page online-data services] in windPRO for more than a decade. 
* '''Built on open standards''': EMDAPI is a [https://en.wikipedia.org/wiki/Representational_state_transfer REST] based service that implements the [https://swagger.io/specification/ OpenAPI] standard]. 
* '''Available from any development tool''': Access to the climate databases is available from your preferred development platform - C#, R, python, html, java, php, scala and swift. Just use the OpenAPI tools to generate the client software for your preferred platform.

== Access ==
To see more documentation and to access the data-services, please visit the API through the following URL's:

* EMD-API Overview (Wiki) - [https://help.emd.dk/mediawiki/index.php?title=Category%3AEMD-API here].
* EMD-API Main Page (API) - [https://api.emd.dk here].
* EMD-API Climate Data UI (API) -  [https://api.emd.dk/climate-data/ui/ here].

Any technical questions on our climate databases can be addressed to our Senior Technical Specialist - Morten Lybech Thøgersen: [mailto:mlt@emd.dk mlt@emd.dk].

== Usage Constraints and Restrictions ==
Each API-access-token provides a full, virtually unrestricted access to the mesoscale and climate time-series datasets. So, even though data access is ‘flat rate’ and without any limitations on the number downloads or number of climate-variables – it is not permitted to do parallel downloads from the same API-token. This is to ensure enough bandwith and a good experience for all API users. This restriction is - from november 2022 - enforced through "rate limiting" - allowing up to 20 API-data-requests per running 10-minute interval. If you exceed this limit, a "''HTTP ERROR 429: TOO MANY REQUESTS''" is shown until below the limit again. Only the following endpoints are part of the rate-limiting procedure:
* /order/
Our recommended approach for API-processing and download is as from the pesudo-code below:
<pre>
prepare list of nodes to download (from "coverage checks")
loop through list:
  request to download point
  wait at minuimum 30 seconds for api-server to finalize data-processing (until status = "success")
  download node from provided url
</pre>

== Data Model - Climate Data Service ==
The EMD climate-data API-services is documented in a REST based API using the OpenAPI Specification. You can view the interfaces and download the interfaces as json or yaml, [https://api.emd.dk/climate-data/ui/ here]. The service provides the following functionality:

* ''List Datasets'': Full list of available datasets with their ID’s, descriptions and documentation liks (URL's). This includes any private datasets connected to your user account.
* ''Locate Data-Nodes'': Request locations of several nearby data nodes from a specific dataset given a latitude-longitude location.
* ''Update Dates'': Information of when a dataset is updated
* ''Place Order'':  Time-series data for any location whith any dataset (from latitude-longitude location). You can decide which period to download.
* ''Order Status'': Request progress for an order - and recieve the download URL for the order.

== Python - Installation == 
[[Image:EgmondAanZee.jpg|thumb|400px|right|Data nodes near the Egmond Aan Zee Offhore ]]The simplest way to use the EMDAPI with python is to install the client software in a virtual environment. If you are using CONDA or [https://docs.conda.io/en/latest/miniconda.html MINICONDA], we recommend that you create a new virtual environment and use a recent 3.x version of python. When the virtual environment is created, then activate the environment.

''Open your Anaconda Prompt. Copy-paste the following lines:''
<pre>
conda create -n emdapiclient python=3.8.5
conda activate emdapiclient
</pre>

Install the required packages needed in order to do data-science and use the examples provided within the jupyter notebooks. We have have validated this setup using specific package versions (used in the commands below). 

''In the  Anaconda Prompt, copy-paste the following lines, one by one:''
<pre>
conda install -c conda-forge pandas=1.1.0 numpy=1.19.1
conda install -c conda-forge matplotlib=3.3.1 basemap=1.2.2 basemap-data-hires=1.2.2 
conda install -c conda-forge jupyter=1.0.0 ipykernel=5.3.4 
conda install -c conda-forge python-wget
</pre>

Download the [https://help.emd.dk/mediawiki/images/7/73/20201206_python_client_generated.zip zipped-file] holding the OpenAPI python client. <br>
Unpack the file and install it within your virtual environment:

''In the Anaconda Prompt: Move to the folder, where you have unpacked the zipped file. Copy-paste the following line:''
<pre>
python setup.py install
</pre>

Make sure that the new emdapi virtual enviroment (python-kernel) is available to be used with jupyter-notebook environment:
<pre>
python -m ipykernel install --user --name=emdapiclient
</pre>

== Python - Jupyter Notebooks for Demonstration and Test ==

In order to test your setup and learn to use the EMDAPI, we suggest that you download the jupyter-notebook examples that we have created - [https://help.emd.dk/mediawiki/images/c/c5/20201206_EMDAPI_JupyterNotebooks.zip here].<br>
Unpack the zip files and run the command below in your terminal or command-shell.<br>
If jupyter prompts for you to select another python-kernel, then select the emdapiclient kernel (may also be selected directly from the 'Kernel' drop-down menu).

''In the Anaconda Prompt: Move to the folder, where you have saved the jupyter-notebook examples. Copy paste the following line to open jupyter notebook from where you can open the examples.'' 
<pre>
jupyter notebook
</pre>

Within the internet-browser (and jupyter user-interface), run select the notebook file (*.ipynb). <br>
Then work your way through through each example provided. Each jupyter notebook holds a separate topic:

# ''emdapi_availability.ipynb'' Demonstration of login to the system, then check which datasets are available at a specific location.
# ''emdapi_correlation.ipynb'': Correlation analysis (R2), check which climate datasets correlates the best against the Egmond Aan Zee data
# ''emdapi_random.ipynb'': Download climate data from any part of the world.

== Client Software Other Languages and Tools ==
REST and OpenAPI is easily consumed from a lot of software tools. It is perfectly possible that your preferred language is supported. OpenAPI works well with languages such as - but not limited to - C#, R, python, java, php, scala and swift. Just download the YAML or JSON service description and use the [https://editor.swagger.io/ Swagger Editor] or [https://github.com/OpenAPITools/openapi-generator OpenAPI Generator] to generate the client libraries for your preferred software. Then you are ready to integrate towards your preferred systems and workflows.

To generate the client libries yourself - one possible process is to:

# Download the OpenAPI (openapi.yaml or openapi.json) description files - [https://api.emd.dk/climate-data/openapi.yaml here-yaml] or [https://api.emd.dk/climate-data/openapi.json here-json]
# Load it into the swagger editor - [https://editor.swagger.io here]
# Choose to "Generate Client" from the drop-down menu within the swagger editor.