 
 
4 EMD Cluster Services 
 
  EMD CLUSTER SERVICES ....................................................................................................................... 2 
4.1  Introduction to EMD CLUSTER SERVICES ............................................................................................ 2 
4.2  WAsP-CFD Cluster Calculation ............................................................................................................... 3 
  Model Overview ........................................................................................................................... 3 
  System Requirements.................................................................................................................. 4 
  Setup of a WAsP-CFD Calculation .............................................................................................. 4 
4.3  EMD-WRF Mesoscale cluster calculation ............................................................................................... 7 
  Setting up calculation in windPRO ............................................................................................... 8 
  Note about turbulence from WRF .............................................................................................. 10 
  Price structure and Meso-Credits .............................................................................................. 11 
  Validation and added information .............................................................................................. 11 
 
 
   

Introduction to EMD CLUSTER SERVICES  2

EMD CLUSTER SERVICES

4.1 Introduction to EMD CLUSTER SERVICES

At EMD we installed a High-Performance Computer (HPC) cluster in 2013 and again in 2018. This cluster gives

windPRO users access to run calculations that cannot be run or would be too time consuming to run on standard

PCs. The computing power of the HPC cluster at EMD is comparable to 2500 standard laptops. Figure 1 shows

part of EMD’s HPC cluster.

At the moment EMD-Cluster services includes two services available via windPRO:

▪ WAsP-CFD flow calculations:

▪ Run high resolution wind flow CFD model from DTU (Ellipsys3D

)

▪ On-demand WRF mesoscale model calculations:

▪ Create long-term time series using the Weather Research Forecast (WRF

) model

Figure 1 - Part of HPC Cluster installed at EMD

Operating and maintaining an HPC cluster is a complex task and running the advanced models requires a detailed

setup of many software components. The windPRO user does not have to worry about all this complexity, as all

calculations are easily set up in windPRO. Once a calculation is set up it is sent to the HPC cluster via the internet

and when the calculation is done the user will be notified via e-mail and the results can be downloaded and used in

windPRO.

An HPC cluster is a large investment, partly due to the hardware itself but also partly because of the power

consumption and cooling requirements in the data centre. Therefore, the user must pay all services separately and

not as part of a windPRO module. This is handled by purchasing “cluster credits” at EMD. The straightforward

approach is that you purchase the credits you need when you need them (or in a bunch for later use). You can also

subscribe to packages where you automatically receive new credits e.g. every month. For further details regarding

pricing please refer to: http://www.emd.dk/windPRO/wasp-products/wasp-cfd/subscription/

The following sections will describe how to use the EMD Cluster services in windPRO.

http://www.wasp.dk/Software/WAsP-CFD/Flow-model

http://www.wrf-model.org/index.php

WAsP-CFD Cluster Calculation  3

4.2 WAsP-CFD Cluster Calculation

The application of wind energy in very complex terrain requires

accurate procedures for deriving reliable estimates of the wind

resource. Traditionally, the modelling-workhorse for the wind

analysts has been the linearized flow models, with WAsP as the

industry standard.

However, recent years have seen the development of non-linear

CFD-models dedicated for wind energy purposes. It is now

possible to use these non-linear models to obtain fairly reliable

alternative estimates in the energy yield assessments. If the

analyses are done the right way - such as ensuring grid-

independent results - the non-linear CFD analysis can render

results that can be documented to have - in general - a lower

uncertainty than what is obtained by the linear models.

Historically, CFD have had two main drawbacks:

▪ CFD requires highly trained users

▪ CFD requires large computational resources in order to minimize model- and numerical errors

WAsP-CFD, however, has its own unique solutions to these problems:

▪ The CFD computations are fully automatic and the setup is similar to normal WAsP calculations

▪ Since the actual CFD calculations runs on a remote supercomputer the user doesn’t have to worry about

computational resources and doesn’t have to compromise between quality and time to run the calculation

Model Overview

The basis of the traditional WAsP model is a superposition of four sub-models

(see the figure to the right):

1) a model to calculate terrain speed-ups (IBZ),

2) a model to calculate the effect of roughness transitions,

3) a model that predicts the vertical flow perturbations due to non-neutral

atmospheric stability,

4) a model to predict reductions caused by obstacles.

In the WAsP-CFD solution flow perturbations of model components 1) and 2) are

replaced by the combined terrain and roughness flow perturbations predicted by

the non-linear Ellipsys3D CFD solver running on EMD’s cluster. The WAsP model

components 1) and 2) assume neutral stability; this is also the case for the flow

perturbations predicted by the WAsP-CFD counterpart. The CFD flow

perturbations are relative to an upstream average roughness named 'reference'

or 'mesoscale' roughness which defines the inlet profile for the CFD solver for

each of the 36 model sectors.

Execution of the model is fully automatic, i.e. the WAsP-CFD model takes care of

setting up the terrain model based on your terrain model, to make a calculation

mesh, to apply proper boundary conditions, to run the analysis to obtain results

at a sufficient convergence criteria and finally to submit the results (speed-ups and wind-veer or back) to you for

further analysis and use with your own wind data.

An overview of some of the automatic modelling steps is outlined in the figures below.

WAsP-CFD Cluster Calculation  4

Figure 2 - Height contours user specified and “massaged” by WAsP CFD

You can read more about WAsP-CFD here:

http://www.wasp.dk/-/media/Sites/WASP/Products/CFD/WAsP-CFD-A-new-beginning-in-wind-resource-

assessment.ashx?la=da

The actual flow model used in WAsP-CFD is described here:

http://www.wasp.dk/waspcfd#flow-model

System Requirements

WAsP-CFD does not require any specific hardware configuration as the computationally demanding part of the CFD

analysis is made on EMD’s HPC cluster. An internet connection is required to upload the terrain and roughness

model and download the flow-analysis results. In order to keep your terrain model safe, all communication from

your PC to the HPC cluster is via a highly encrypted and protected HTTPS-connection.

Figure 3 - Illustrates flow of data to and from the client

In terms of software licenses, you will need the following in order to operate the WAsP-CFD model in windPRO:

▪ WAsP 11 (or higher)

▪ WindPRO 2.9 (or higher) with the MODEL module (or other relevant energy modules)

▪ Calculation credits

Calculation credits can be purchased from the EMD Sales department, see more in the section below. A credit

covers a tile (calculation area) of 2 km x 2 km.

If you have older versions than WAsP 11 installed on your computer, make sure the version 11 or newer is selected

before starting your CFD jobs.

Setup of a WAsP-CFD Calculation

The WAsP-CFD analysis is very similar to the WAsP analysis and it requires the same terrain data but to a larger

extension, at least 20km from the area of interest in all direction. You need a site data object to link the terrain data

to the CFD calculation, similar to a 'classic' WAsP calculation. The purpose of the Site data object should be ‘For

use in CFD calculation’.

WAsP-CFD Cluster Calculation  5

Figure 4 - Purpose of Site Data object when used for WAsP-CFD

4.2.3.1 Terrain tab - Elevation / Orography model

Elevation data are required within a range of at least 20km from the site. If needed, existing elevation data can be

extended directly via the line or the grid objects.

When height data is available in a grid object, it will be converted automatically to lines when the CFD calculation

is run (the current version of WAsP-CFD needs data as lines). The setup of the elevation grid object should be with

a contour separation of 10 m or 20 m in the outer zones as this gives a sufficient degree of detail, but in the local

zone (about 8 km around the site), the contour separation should preferably be 5 m if justified by the underlying

data. The contour separation is defined under the ‘WAsP-setup’ tab of the grid object as shown below.

Figure 5 - Setting elevation data in an elevation grid object

4.2.3.2 Terrain tab - Roughness model

The surface roughness model must be defined as lines and should cover an area not less than 20 km from the area

of interest.

4.2.3.3 CFD-areas (Tiles)

The model output from the cluster part of the WAsP-CFD calculation is returned as area(s) or tile(s) each 2km by

2km in size. It is important that all the turbines, measurement mast(s) or resource areas are covered by one or more

CFD areas. There is no limitation on how the areas/tiles may be placed, they may be aligned by edges, separated

by some distance or even overlapping.

Figure 6 - Five WAsP-CFD tiles defined on a site

WAsP-CFD Cluster Calculation  6

The CFD-areas are created from the site data object either directly at the tab ‘Resource / CFD Area’ by pressing

the ‘Add’ button or the ‘Auto create’ button. The latter requires a WTG Area to define the area of interest. The

coordinates of the centre of tile and the name are editable (but can also be changed from the map once the site

data object is closed).

Pressing the ‘Auto create’ button will create the required tiles to cover a given WTG area object.

Figure 7 - Auto create can be used to generate tiles from WTG area object

Click on OK to close and create the site data object.

4.2.3.4 Edit CFD tiles on the map

Once the Site Data Object with CFD purpose is closed, right-click to bring it into ‘Edit mode’ (see below).

Figure 8 - Use Edit mode for CFD Areas to manipulate tiles directly on the map

When the site data object is in Edit mode it is possible to:

▪ Move the individual CFD-squares by dragging in the center of the square.

▪ Add new areas by right clicking the site data object and choosing 'Add CFD area'.

▪ Edit the properties of an area by right clicking the center of the area and choosing “Edit area properties”.

Only the coordinates of the center and the name of the area can be changed.

▪ Delete an area by right clicking the center of the area and choosing “Delete”.

When you are done with the editing the CFD areas from the map, it is necessary to right-click on the Site Data

Object and uncheck “Edit mode for CFD areas”.

4.2.3.5 WAsP-CFD calculation

Assuming you have the relevant number of CFD calculation credits, once the data has been prepared, the CFD

calculations has to be started on the HPC cluster. In order to this follow this procedure:

▪ Create a new Model: WAsP-CFD calculation from the Energy tab.

▪ Name the calculation and select the site data object containing your CFD-areas.

EMD-WRF Mesoscale cluster calculation  7

▪ Go-to the tab-sheet named 'Calculations'.

▪ Press the Login button.

▪ Choose which areas to include in the CFD-analysis (by checking the boxes in the selected column).

▪ Press the 'Start selected' button

▪ Confirm that you want to start

▪ The job is started on the server by uploading the data (orography, roughness and tile areas)

▪ Once the job is started you can see the estimated completion time

▪ Press Ok to close calculation

▪ Save your windPRO project

After this you can close windPRO or continue any other work in windPRO while the calculations are running on

the cluster.

4.2.3.6 Download results

Once each area has been calculated, an e-mail will be sent to the user simply saying “Job completed OK”. The e-

mail also contains a single page PDF report on the calculation results. To download the results:

▪ Start windPRO and load the project

▪ Open the CFD calculation and go to the 'Calculations' tab

▪ Login to the cluster (this process is automatic and you do not need to be supplied with any login details)

▪ Make sure “Add as result layer” is checked

▪ In the table you will be able to see that the 'Download Results' button is now available.

▪ Click the button to download the results and add them as result layers.

The CFD result file(s) will be placed in a sub-directory in the project folder named '/OnlineCFDResults'. The

extension of the file(s) is .CFDRES.

4.3 EMD-WRF Mesoscale cluster calculation

High quality mesoscale data is becoming a more and more important factor in wind energy calculations. The data

has a wide variety of uses in wind farm planning, such as:

▪ Long-term correcting measurements

▪ Initial siting purposes

EMD-WRF Mesoscale cluster calculation  8

▪ Production follow up

▪ Estimating icing losses

▪ Estimating shadow flicker using solar irradiation parameters

▪ Evaluate the atmospheric stability

For several years windPRO has provided free online data such as MERRA, NCAR, CFSR etc. which has been used

for long-term correction in MCP. The spatial resolution of those datasets is in the range of 70-150 km so even

though they in some areas have a good correlation with regards to wind speed they cannot capture mesoscale

effects in the wind climate which occur on the spatial scales of few kilometers.

In windPRO 2.9 we introduced the mesoscale dataset ‘EMD-ConWx Europe’ which covers central parts of Europe

and has a spatial resolution of approx. 3 km. The dataset is a pre-run dataset which means that the data have

already been processed and can be downloaded directly like other online datasets in windPRO, with the exception

that an annual subscription is required. The EMD-ConWx dataset is processed using the Weather Research

Forecast (WRF) model on the HPC cluster at EMD. The dataset uses ERA-Interim global boundary data from

ECMWF.

Since then, several more EMD-WRF datasets have been made available for various locations around the World

(e.g. Middle East, Indonesia, South Africa, South Korea and India). The goal for EMD is to provide more of these

pre-run datasets in the future, but until more of them are available we provide a service for windPRO users that will

allow them to run on-demand WRF mesoscale calculations anywhere in the world. So instead of a pre-run dataset

like ‘EMD-WRF Middle East’, the on-demand service is run as required for the limited area surrounding a single

point of interest.

Besides being worldwide the on-demand solution also has a number of other advantages compared to the pre-run

EMD-ConWx data:

▪ It uses the newest version of the WRF model

▪ It has a better background roughness model

▪ It has better solar irradiation results

▪ It has turbulence information for all years simulated

▪ More climate parameters are available

The pre-run EMD-ConWx data sets were in 2019 replaced with new EMDWrf data sets, based on ERA5 and newer

WRF model. The areas covered has changed slightly, see Pre-Run Time Series for updated information of available

pre-run data sets.

For the on demand WRF calculations, improved options are introduced:

• ERA5T as input, has only 1 month delay compared to ERA5 with at least 3-month delay

• Special WRF model setup for Icing calculations, which e.g. can deliver icing loss maps

Figure 9 Example of an Icing loss calculation with output of icing parameters as result layer.

Setting up calculation in windPRO

Running an EMD-WRF on-demand calculation is easy and simple - you only need to define:

EMD-WRF Mesoscale cluster calculation  9

▪ Geo Location

▪ Model configuration

▪ Period

Once you have bought sufficient meso-credits, you can start a new calculation by creating a Mesoscale Data

calculation in windPRO from the Climate section:

This will open the calculation setup window.

Figure 10 Wrf on demand setup.

The WRF model run is setup at the main tab.

The position for the wanted data point is chosen, can be pointed out on map. Only one point is calculated.

Model configuration:

Multiple configurations are available:

Figure 11 Model configurations available for WRF-on-demand.

SRTM is used as elevation data for all variants.

GlobCover is used for roughness

Meteorological data is offered in several variants:

• ERA5T

• ERA-Interim

• MERRA-2

• CFSR/CFSv2

• ERA5

• ERA5 Icing

EMD-WRF Mesoscale cluster calculation  10

For Icing, a special WRF setup (Thompson scheme) is used dedicated for icing calculations.

Time period for the time series output can be selected. There is an option to check “latest available data” which

gives the newest possible month to include based on the selected configuration. The meteorological data behind

the model run are typically updated monthly, some like ERA-Interim are although not updated after August 2019.

Note that the longer period, the more credits is needed, and some models needs more credits than other.

For Icing calculations, minimum 10 winter seasons must be included to get the extended statistical output, while

this requires at least 10 years to make a decent statistical basis.

Once the data are prepared for the WRF model it is set up to run the domain based on the location given by the

user in windPRO. The model will then run for the period defined by the user. The current spatial resolution is 3 km

and the model simulates every 9 seconds in time. Each hour the current state of the model is extracted and

converted into a time series. Besides the time series data, the result will also include the terrain (orography and

roughness) from the mesoscale simulation to be used in downscaling calculations in windPRO.

The entire process from downloading boundary data to getting end result as time series is fully automated and not

something the user has to worry about.

Once you have set up everything, it is time to start the calculation on the computer cluster at EMD. This is done

by pressing the “Start on Server” button. This will give you the following window.

In this window you can see the meso-credits that you have purchased and select in which order they should be

used. In the window you can also see how many credits that will be used for the calculation (1 meso-credit in this

case). Once you are ready press the ‘Ok’ button to go ahead and start the calculation. You will see one more

confirmation dialog before the calculation is started. Simply click ‘Yes’ if you are sure you want to start the

calculation.

When the calculation has started you will see the ‘Calculations’ tab that shows you the calculation you just started

and also show the estimated completion time for the job. During busy periods at the EMD cluster there will be a

longer waiting time. Click the ‘Ok’ button and save the project.

Once the job has completed you will receive an e-mail confirming that the results can be downloaded in windPRO.

Open the project in windPRO and find the MesoScaleData calculation and open it. On the “Calculations” tab you

should see the calculation you started and the button “Download results” should now be enabled. Click the button

to download the results from the server. After the download completes the result is added as a Meteo object in the

windPRO project. For Icing calculations also a result layer is added where different parameters, like the expected

AEP loss, can be shown for a 10 x 10 km area.

4.3.1.1 Parameters in the output time series

The main results included in the time series from EMD-WRF on-demand are:

▪ Wind speed/direction and turbulence in 10, 25, 50, 75, 100, 150 and 200 meters.

▪ Temperature in 2 and 100 meters.

▪ Detailed solar irradiation

▪ Surface pressure, humidity, precipitation, visibility, cloud cover, etc.

Note about turbulence from WRF

When running the WRF model, there is two different approaches regarding turbulence:

 
EMD-WRF Mesoscale cluster calculation  11 
 
© EMD International A/S   •    www.emd.dk   •   windPRO 4.0   •   September 2023 
 
1.  Multiplying the WRF modeled turbulence with Square root(2). 
2.  Use “as is” 
 
The 1. Is used in EMDConWx and WRF on demand in present implementation. 
The 2. Is used for EMDWrfEU+ dataset and other new pre-run data sets. 
 
The reason for the two approaches is that 1. were based on handed over experience, while 2. Is based on validation 
against internal EMD mast data base, that in average show the correction is not needed. There are although still 
doubts what is the most correct. Offshore validation examples indicates that there is a need for the correction 
mentioned in 1. The problem is that the WRF modeled turbulence in general is quite uncertain while turbulence 
often is a very local phenomenon. We have therefore added an option to scale the turbulence when it is used in 
e.g. a PARK calculation and recommend for offshore sites to scale with Square root(2) when EMDWrf pre-run data 
sets are used. For all locations it is recommended to compare with local measurements if available and take the 
scaling decision from this. 
 
  Price structure and Meso-Credits 
No particular modules are required in order to run EMD-WRF on-demand calculations, though the data could be 
used in modules such as METEO and MCP. Instead, the calculations are paid for when you request EMD to perform 
the calculations and the price depends on how long a time series you request. We introduce the concept of a meso-
credit. A single meso-credit will allow you to get 1 month of hourly time series data from one point for a “standard 
WRF”. So for a 10-year time series the cost will be 12 (months) * 10 (years) = 120 meso-credits. The meso-credits 
are  purchased  by  contacting  sales@emd.dk  or  online  at  http://www.emd.dk/windpro/online-ordering/.  Find  the 
button in right part of the screen. 
 
 
Figure 12 Present pricing for Meso credits. 
 
  Validation and added information 
Our experience is that either the ERA5 or ERA5-T datasets generally result in the best performance when correlating 
wind speeds against those obtained from local meteorological towers. ERA5 was used as boundary in the ‘EMD-
WRF Europe +’ dataset with excellent performance, see our Technical Note 7: EMD-WRF Europe+ Validation. 
 
More information about the EMD-WRF On-Demand model performance can be found here: Technical Note 6: EMD-
WRF On-Demand and details on parameters and availability are available at our Wiki Page: On-Demand Datasets.  
 
For Icing details see: EMD-WRF On-Demand ICING - Wiki-WindPRO 
 