Calibration or reliability of forecasts is the ability of a model to
correctly identify its own uncertainty in making predictions. In a model
with perfect calibration, the observed data at each time point look as if
they came from the predictive probability distribution at that time.
Equivalently, one can inspect the probability integral transform of the
predictive distribution at time t,
$$
u_t = F_t (x_t)
$$
where \(x_t\) is the observed data point at time \(t \textrm{ in } t_1,
…, t_n\), n being the number of forecasts, and \(F_t\) is
the (continuous) predictive cumulative probability distribution at time t. If
the true probability distribution of outcomes at time t is \(G_t\) then the
forecasts \(F_t\) are said to be ideal if \(F_t = G_t\) at all times t.
In that case, the probabilities \(u_t\) are distributed uniformly.
In the case of discrete nonnegative outcomes such as incidence counts,
the PIT is no longer uniform even when forecasts are ideal.
In that case two methods are available ase described by Czado et al. (2007).
By default, a nonrandomised PIT is calculated using the conditional
cumulative distribution function
$$
F(u) =
\begin{cases}
0 & \text{if } v < P_t(k_t - 1) \\
(v - P_t(k_t - 1)) / (P_t(k_t) - P_t(k_t - 1)) & \text{if } P_t(k_t - 1) \leq v < P_t(k_t) \\
1 & \text{if } v \geq P_t(k_t)
\end{cases}
$$
where \(k_t\) is the observed count, \(P_t(x)\) is the predictive
cumulative probability of observing incidence \(k\) at time \(t\) and
\(P_t (-1) = 0\) by definition.
Values of the PIT histogram are then created by averaging over the \(n\)
predictions,
$$
\bar{F}(u) = \frac{i = 1}{n} \sum_{i=1}^{n} F^{(i)}(u)
$$
And calculating the value at each bin between quantile \(q_i\) and quantile
\(q_{i + 1}\) as
$$
\bar{F}(q_i) - \bar{F}(q_{i + 1})
$$
Alternatively, a randomised PIT can be used instead. In this case, the PIT is
$$
u_t = P_t(k_t) + v * (P_t(k_t) - P_t(k_t - 1))
$$
where \(v\) is standard uniform and independent of \(k\). The values of
the PIT histogram are then calculated by binning the \(u_t\) values as above.