An ensemble using the members WRF KF, WRF BMJ, ETA KF, and ETA BMJ is compared to an ensemble using the members WRF KF, WRF BMJ, NMM KF, and NMM BMJ. A total of 31 120 hour integrations were performed at 00 UTC between the dates 01 September and 11 October, 2005. Plots of the forecast vs. observed rainfall from all the ensemble members along with plots of the forecast scores for all of the individual forecasts are archived at the URL:

http://mtarchive.geol.iastate.edu/verification/

Summary of Results:

** Note: Only results of the Day 1 forecasts were examined

ROC SCORES

A comparison of the average ROC Scores from the ensemble using WRF NMM and the ensemble using Workstation Eta revealed that neither ensemble displayed a significant advantage. A plot of the average ROC Scores from both ensembles is displayed below:

However, it did appear that the NMM KF did have an advantage over the ETA KF. A comparison of the ROC Scores from these individual ensemble members is displayed below:

Average ROC Scores from the BMJ members shows that the ETA seems to perform better at the lower thresholds, while the NMM seems to perform better at the higher thresholds. The average ROC Scores from the BMJ members are displayed below:

BIAS SCORES

The bias scores reveal some unexpected results. A comparison between the BMJ member of the ETA and NMM shows that the NMM has lower bias scores at the lower thresholds and much higher bias scores at the highest thresholds. The NMM BMJ member is behaving more like we would expect a KF member to behave. A comparison between the KF member of the ETA and NMM shows that the NMM has higher bias scores at the lowest thresholds (except 0.01 inches) and lower bias scores at the highest thresholds. The NMM KF is behaving more like we would expect a BMJ member to behave. An obvious explanation is that the NMM BMJ and NMM KF members were accidentally switched in the calculations. I checked for this and it was not the case! The plots showing the aformentioned results are displayed below:

EXAMPLES

- Day 1 Forecast vs. Observed Rainfall: 12 September 2005

WRF NMM

Workstation ETA

- Day 1 Forecast vs. Observed Rainfall: 02 September 2005

WRF NMM

Workstation ETA