Evaluating the performance of probabilistic forecasts of univariate and multivariate quantities
 (NOAA) and (CIRES)
Date and time:Â
Friday, December 5, 2014 - 3:00pm
³¢´Ç³¦²¹³Ù¾±´Ç²Ô:Ìý
ECCR 245
´¡²ú²õ³Ù°ù²¹³¦³Ù:Ìý
Many statistical methods in disciplines such as economics, atmospheric sciences, epidemiology, and environmental modeling are concerned with forecasting and interpolating uncertain, unknown, or partially known quantities. To communicate the associated prediction uncertainty, probabilistic forecasts taking the form of probability distributions should be provided rather than just a single 'best' prediction.
To assess the quality of probabilistic forecasts, both diagnostic, qualitative tools, and quantitative performance measures have been proposed. In the first part of the talk we review the notions of 'calibration' and 'sharpness' which together characterize the quality of a probabilistic forecast. We give a brief overview over tools that have been developed to verify those two properties for forecasts of univariate quantities. Those tools will include probability integral transform (PIT) histograms and so called proper scoring rules, and will be illustrated in a data example with probabilistic 80m wind speed forecasts at five major wind park locations in Colorado.
In the second part of the talk we consider probabilistic forecasts of multivariate quantities, which are the subject of ongoing research with much fewer verification tools being available so far. We discuss extensions of the PIT histogram to the multivariate case, and present a new class of multivariate scoring rules that is based on the geostatistical concept of variograms, and is applicable also in the case where the multivariate forecast is represented by a - possibly rather small - sample from the predictive distribution. This class is compared with two established multivariate scoring rules in in a number of examples with simulated observations and forecasts, and finally applied to the wind speed forecast example from the first part of the talk.