Session of the wmo-ioc joint Technical Commission for Oceanography and Marine Meteorology (jcomm) agreed that it would be logical to transform the wmo wave Programme into the jcomm wind Wave and Storm Surge Programme
Download 1.37 Mb.
|
- Navigate this page:
- 6. OPERATIONAL FORECAST VERIFICATION
- PORT SIZE CORR MEAN S.D. RMSE MAX ERR DATE
- 6.2 Routine Verification of the Danish Storm Surge Model
- 6.3 Routine Verification of the German BSH Storm Surge Model
- 6.4 Routine Verification of the Netherlands DCSM98 Storm Surge Model
- 6.5 Total Forecast Verification – an example from the UK Coastal Monitoring and Forecasting (UKCMF) Service
- Alert level reached Alert issued
The system in the NetherlandsThe system in the Netherlands uses input from the ECMWF’s Ensemble Prediction System (EPS). This has a deterministic forecast and a lower-resolution ensemble with a control forecast and 50 perturbed members. All members are used to drive the storm surge model. The output is mainly used for skew surge forecasts for locations at the coast of the Netherlands. The individual forecasts are transformed into probability forecasts with a calibration derived from an earlier winter season of forecasts. Figure 5.6 gives an example a surge forecast based for Vlissingen in the south of the country. Skew surges from all individual members are shown together with calibrated probabilities. Orange asterixes are the observed surges.  
Figure 5.6: Probability surge forecast, based upon the ECMWF ensemble, see text. Figure 5.6: Probability surge forecast, based upon the ECMWF ensemble, see text. 6. OPERATIONAL FORECAST VERIFICATION This chapter reviews the various approaches to forecast verification that are applied to operational surge models. These methods are routine analyses, carried out on a permanent or periodic basis, and involve the comparison of model output with some observed quantity. The prime purpose of verification is to monitor the consistency (in time or space) of model outputs, and to highlight the steps that may be taken to improve the quality of a forecast. In practice, the most commonly compared variable is total sea level or sea level residual where the observational data are obtained from tide gauge networks. Although not a routine choice, the verification of horizontal currents is a feasible proposition using data from modern instruments such as Acoustic Doppler Current Profilers (ADCP’s). Indeed, verification of model performance away from the coast (i.e. places other than where tide gauges are sited) is an area of monitoring that demands further investment in order to ensure optimum model performance. In the future, satellite observations of sea level from altimetry may be available for verification but that is not practicable at the time of writing. When comparing observed and modeled sea levels, one must be mindful of the differing nature of the two datasets. The majority of finite difference surge models are based on a C-grid (Arakawa and Lamb, 1977) which implies that the model elevation point is half a grid cell distant from the modeled representation of the physical coastline; for medium or coarse resolution models this may necessitate extrapolation of the model variable. Since the model output represents average conditions over several time-steps then some time-averaging of observations may also be required. As noted in Chapter 4 of this volume, in many operational systems a modeled surge (plus tide-surge interaction) is generated by running the simulation with, and then without, meteorological forcing and subtracting the two. Unfortunately, as well as representing surge and tide-surge interaction, the observed non-tidal residual may also contain errors of instrument timing and harmonic prediction (Horsburgh and Wilson, 2007). As shown in that paper, small errors of phase in harmonic predictions can cause significant oscillations in the calculated residual. The annual variability of tidal constituents is discussed in Pugh (1987, p133). Although tides can be predicted to great accuracy, the combined error over 60-100 constituents can amount to several centimetres of amplitude, or phase errors equating to several minutes. These errors are typically largest in regions of high tidal amplitude. Bearing all this in mind, it is good practice to use the word “surge” only when implying a genuine meteorological contribution to sea level. Comparisons with observational data will usually take the form of a set of statistics, calculated monthly. The parameters of interest for sea level are the correlation, mean error (which can reveal model biases), maximum error, and root mean square (RMS) error. More concise verifications can be obtained via a specified skill score (e.g. Bogden et al., 1996), which normally takes the form of a ratio between the instantaneous model error and the departure from a reference, or background value:  (6.1)
where Oi denotes the series of observed values, Mi the modeled value and Ci the background (often climatic) value at that location. Comparisons of several variables (e.g. surface elevation and currents) can be combined in a weighted sum, or cost function. For a thorough guide to oceanographic skill assessment see Lynch and Davies (1995). If possible, the process of verification should be applied to the entire forecast suite: this includes the meteorological forcing and all components of the marine forecast delivered to the customer of the operational system. There is a long history of forecast verification in meteorology, and an excellent overview can be found at http://www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html. All of these techniques are equally applicable to storm surge forecasting. Examples of verification procedures from several regional operational models are now given below . 6.1 Routine Verification of the UK Operational Storm Surge Model The National Oceanography Centre (NOC) develops and maintains the tide-surge models used to forecast storm surges around the UK coast, on behalf of the UK Environment Agency. Archived port data, from the tide gauge network in Figure 6.1, are transferred back to NOC at regular intervals in order to assess the model’s performance. Monthly time series plots for each port are available on the internet (www.pol.ac.uk/ntlsf/engineers/surgemonthlyplots). Additionally, simple statistics are calculated based on the difference between hourly model and observations (model residual – observed residual). From this, mean, standard deviation, correlation coefficient, RMS error, maximum error and the time at which that occurred, are calculated. Importantly, these parameters are calculated for the hour closest to model high water for each month, as these have the most practical relevance for flood forecasting. 
Figure 6.1: The UK national tide gauge network (see http://www.pol.ac.uk/ntslf/networks.html for more details) An example statistical table is given in Table 6.1, where the column headings are: SIZE: Sample size (i.e. where there exists both a model and observed value) CORR: Correlation coefficient MEAN: The arithmetic mean of the series S.D.: The standard deviation from the mean RMSE: The root mean square (RMS) error MAX ERR: The maximum difference between model and observation occurring in the series Table 6.1: Surge model statistics for the UK model based on hourly values for January 2005.
6.2 Routine Verification of the Danish Storm Surge Model The Danish storm surge model (Mike21) was developed by the Danish Hydraulic Institute (DHI). Tide gauge data are retrieved from the DMI data base, where all on-line Danish tide gauge data are routinely stored. The data providers are the DMI, the Coastal Authorities, the Royal Danish Administration of Navigation and Hydrography, and a number of local authorities. Stations are indicated in Figure 6.2. 
Figure 6.2: The Danish tide gauge network Sea level predictions are compared with tide gauge data on an annual basis, and monthly forecast errors are calculated as annual running means. The mean error, RMS error, explained variance and correlation coefficient are calculated for each station. This is done as a function of forecast range. Another verified parameter is the peak error (alternatively skew surge) - the difference between the highest predicted and the highest observed sea level during a storm. Finally, a ‘quality’ parameter, Q, is defined as the mean absolute peak % bias of the 3 highest peaks at 18 selected stations. Q is calculated once a month, as a running 12-month average. These results are shown in Figures 6.3 and 6.4.  
Figure 6.3: RMS error statistics for all Danish stations (2006) 
Figure 6.4: Quality factor for the Danish operational model (2003-2006) 6.3 Routine Verification of the German BSH Storm Surge Model High and low water predictions for the German Baltic and North Sea stations are evaluated on an annual basis. Biases and standard deviations of high water predictions are calculated at all stations. The example in Figure 6.5 is for the port of Cuxhaven, for 2003. Higher bias values are typically found for low water differences because the complex topography of the Wadden Sea and German estuaries has a stronger effect on low water than high water, even in the 1 nautical mile resolution model. The high water and low water model errors (2000-2005) for skew surge from the BSH operational model are shown in Table 6.2 below Figure 6.5: Example frequency distribution of high water errors from the German BSH storm surge model, for the port of Cuxhaven
6.4 Routine Verification of the Netherlands DCSM98 Storm Surge Model The Dutch storm surge model, run operationally by KNMI, is verified against observational data from the Rijkwaterstaat network. Results of the model are routinely evaluated with observed skew surges (the difference between the highest predicted and the highest observed sea level) at the following port locations: Vlissingen, Roompot Buiten, Hoek van Holland, Ijmuiden, Den Helder, Harlingen and Delfzijl. The following examples show 30 day averages of the RMS error, bias and standard deviation of the high water skew surge for Hoek van Holland over the years 1997-2007. It should be noted that the version of the model with data assimilation (Kalman filter – red line) exhibits improved performance; the data assimilation reduces all the error parameters and removes seasonal signals from the verification plots. 
Figure 6.6: RMS (m) of the skew surge (30 day averages) for the DCSM. Colours denote different forecast periods as indicated in the caption. 
Figure 6.7: Bias (m) of the skew surge (30 day averages) for the DCSM. Colours denote different forecast periods as indicated in the caption. 
Figure 6.8: Standard deviation (m) of the skew surge (30 day averages) for the DCSM. Colours denote different forecast periods as indicated in the caption. 6.5 Total Forecast Verification – an example from the UK Coastal Monitoring and Forecasting (UKCMF) Service Ideally the whole forecasting system should be verified, including both the local winds and wind conditions in the generating region of the storm surge, through the modeled surge to the timeliness and accuracy of issued alerts. Depending on how the surge forecasting system is structured, it may be possible for a forecaster to judge that the modeled atmospheric forcing is inaccurate, and to thence modify the predicted surge value. In this case it is useful to verify both the raw model and forecaster values in order to judge the effectiveness of the forecaster intervention. The example verification products in this section come from the 2004-2005 Annual Report of the UK Coastal Mointoring and Forecasting (UKCMF) Service, which is the operational coastal flood warning operation in the UK. The system is judged, annually, on three principal measures – performance measure, skill score and timeliness. The performance measure is an overall metric that assesses the timeliness of the issue of an alert and the significance of the observed sea level in relation to some predefined alert levels (which are prescribed for each region). The principle is - the more significant the event, the larger the reward but also the larger the penalty if no alert was issued (or if it was late). Table 6.3 shows the number of events in each category for the period April 2004 to March 2005. Table 6.3: Events table for all ports in England and Wales from April 2004 to the end of March 2005 that was used to generate the example performance measure.
The performance is measured by multiplying the number of occurrences in a category by the weighting for that category (see Table 6.4) and then summing all components. This actual score is divided by the perfect score to give a percentage figure. For the period shown here the actual performance measure was 87%. Reporting the forecast system performance with a single indicator is often useful when communicating to non-scientific stakeholders (e.g. policy makers, local planners). Table 6.4: The weightings used in the performance measure calculation.
The skill score is calculated from the dataset of events used for the performance measure, but excludes tides that do not reach significant alert level (i.e. those in the no alert/near miss category). Skill scores are typically higher than performance measure. Directory: pub pub -> Project Scoring Template Pre-Construction Phase New Core/Branch Campus Projects pub -> Acm word Template for sig site pub -> The german unification, 1815-1870 pub -> Baltic Olympiads in Informatics: Challenges for Training Together pub -> Preparation of Papers for ieee transactions on medical imaging pub -> Forthcoming meetings and conferences pub -> Asilomar Conference on Circuits, Systems & Computers, Pacific Grove, ca, November 1978, pp. 55 58 pub -> Forthcoming meetings and conferences pub -> Harmonised compatibility and sharing conditions for video pmse in the 7 9 ghz frequency band, taking into account radar use Download 1.37 Mb. Share with your friends:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||