GAS (version 0.3.4.1)

BacktestDensity: Backtest a series of one-step ahead density predictions.

Description

The BacktestDensity() function accepts an object of the class uGASRoll, and returns a list with two elements: (i) the averages Negative Log Score (NLS) and weighted Continuous Ranked Probability Score (wCRPS) introduced by Gneiting and Ranjan (2012), and (ii) their values at each point in time. The wCRPS is evaluated using 5 weight functions, see Details.

Usage

BacktestDensity(Roll, lower, upper, K = 1000, a = NULL, b = NULL)

Value

A list with elements: average, series. The element "average" is a named vector with the averages NLS and wCRSP. The element "series" is a list: the fist element, LS, contains the out-of-sample Log Score (not with the negative sign), the second element, WCRPS, contains a matrix

with the wCRPS series. The columns of this matrix are named: "uniform", "center", "tails", "tail_r", "tail_l", which are associated with the wCRSP with emphasis on: Uniform, Center, Tails, Right tail and Left tail, respectively.

Arguments

Roll

an object of the class uGASRoll.

lower

numeric the lower bound used to approximate the wCRSP by Monte Carlo integration as detailed in Gneiting and Ranjan (2012). This coincides with \(y_{l}\) in Equation 16 of Gneiting and Ranjan (2012).

upper

numeric the upper bound used to approximate the wCRSP by Monte Carlo integration as detailed in Gneiting and Ranjan (2012). This coincides with \(y_{u}\) in Equation 16 of Gneiting and Ranjan (2012).

K

numeric integer representing the number of points used to discretize the wCRPS integral. This is \(I\) in Equation 16 of Gneiting and Ranjan (2012). By default K = 1000.

a

numeric. mean of the normal distribution used in the weight function. By default a = NULL, which means that it is set equal to the empircial mean of the in sample observations.

b

numeric. standard deviation of the normal distribution used in the weight function. By default b = NULL, which means that it is set equal to the empircial standard deviation of the in sample observations.

Author

Leopoldo Catania

Details

The average Negative Log Score (NLS) is computed as the negative of the average of the log scores evaluated during the out-of-sample period. The average weighted Continuous Ranked Probability Score (wCRPS) is computed as the average of the wCRPS evaluated during the out-of-sample period, see Gneiting and Ranjan (2012).
The wCRPS is evaluated using Equation 16 of Gneiting and Ranjan (2012). The weights functions implemented are:

• \(w(z) = 1\): Uniform,

• \(w(z) = \phi_{a,b}(z)\): Center,

• \(w(z) = 1 - \phi_{a,b}(z)\): Tails,

• \(w(z) = \Phi_{a,b}(z)\): Right tail,

• \(w(z) = 1 - \Phi_{a,b}(z)\): Left tail,

where \(\phi_{a,b}(z)\) and \(\Phi_{a,b}(z)\) are the pdf and cdf of a Gaussian distribution with mean \(a\) and standard deviation \(b\), respectively. The label "Uniform" represents the case where equal emphasis is given to all the parts of the distribution.

References

Gneiting T, Ranjan R (2011). "Comparing Density Forecasts using Threshold -and Quantile-Weighted Scoring Rules." Journal of Business & Economic Statistics, 29(3), 411-422. tools:::Rd_expr_doi("10.1198/jbes.2010.08110").

Examples

if (FALSE) {
data("cpichg")

GASSpec = UniGASSpec(Dist = "std", ScalingType = "Identity",
                       GASPar = list(location = TRUE, scale = TRUE,
                                      shape = FALSE))

Roll = UniGASRoll(cpichg, GASSpec, ForecastLength = 50,
                  RefitEvery = 10, RefitWindow = c("moving"))

BackTest = BacktestDensity(Roll, lower = -100, upper = 100)

BackTest$average
}