To be considered reliable, a model must be calibrated so that its confidence in each decision closely reflects its true outcome. In this blogpost we'll take a look at the most commonly used definition for calibration and then dive into a frequently used evaluation measure for model calibration. We'll then cover some of the drawbacks of this measure and how these surfaced the need for additional notions of calibration, which require their own new evaluation measures. This post is not intended to be an in-depth dissection of all works on calibration, nor does it focus on how to calibrate models. Instead, it is meant to provide a gentle introduction to the different notions and their evaluation measures as well as to re-highlight some issues with a measure that is still widely used to evaluate calibration.
Calibration makes sure that a model’s estimated probabilities match real-world likelihoods. For example, if a weather forecasting model predicts a 70% chance of rain on several days, then roughly 70% of those days should actually be rainy for the model to be considered well calibrated
Now, what calibration means more precisely depends on the specific definition being considered. We will have a look at the most common notion in machine learning (ML) formalised in
A model is considered confidence-calibrated if, for all confidences \(c,\) the model is correct \(c\) proportion of the time:
\[\mathbb{P} (Y = \text{arg max}(\hat{p}(X)) \; | \; \text{max}(\hat{p}(X))=c ) = c \;\;\:\: \forall c \in [0, 1] \; ,\]This definition of calibration, ensures that the model’s final predictions align with their observed accuracy at that confidence level
For simplification, we assume that we only have 3 classes as in image 2 and we zoom into confidence \(c=0.7\), see image above. Let’s assume we have 10 inputs here whose most confident prediction (max) equals \(0.7\). If the model correctly classifies 7 out of 10 predictions (true), it is considered calibrated at confidence level \(0.7\). For the model to be fully calibrated this has to hold across all confidence levels from 0 to 1. At the same level \(c=0.7\), a model would be considered miscalibrated if it makes only 4 correct predictions.
One widely used evaluation measure for confidence calibration is the Expected Calibration Error (ECE)
where \(B\) is used for representing “bins” and \(m\) for the bin number, while acc and conf are:
\[\small{ acc(B_m) = \frac{1}{ \mathopen| B_m \mathclose|} \sum_{i\in B_m} \mathbb{1} (\hat{y}_i = y_i ) \;\: \text{&} \;\: conf(B_m) = \frac{1}{ \mathopen| B_m \mathclose|} \sum_{i\in B_m} \hat{p}(x_i) }\]\(\hat{y}_i\) is the model’s predicted class (arg max) for sample \(i\) and \(y_i\) is the true label for sample \(i\). \(\mathbb{1}\) is an indicator function, meaning when the predicted label \(\hat{y}_i\) equals the true label \(y_i\) it evaluates to 1, otherwise 0. Let’s look at an example, which will clarify acc, conf and the whole binning approach in a visual step-by-step manner.
In the image below, we can see that we have \(9\) samples indexed by \(i\) with estimated probabilities \(\hat{p}(x_i)\) (simplified as \(\hat{p}_i\)) for class cat (C), dog (D) or toad (T). The final column shows the true class \({y}_i\) and the penultimate column contains the predicted class \(\hat{y}_i\).
Only the maximum probabilities, which determine the predicted label are used in ECE
We still need to determine if the predicted class is correct or not to be able to determine the average accuracy per bin. If the model predicts the class correctly (i.e. \(y_i =\hat{y}_i\)), the prediction is highlighted in green; incorrect predictions are marked in red:
We now have visualised all the information needed for ECE and will briefly run through how to calculate the values for bin 5 (\(B_5\)). The other bins then simply follow the same process, see below.
We can get the empirical probability of a sample falling into \(B_5\) , by assessing how many out of all \(9\) samples fall into \(B_5\), see \(\mathbf{(\;1\;)}\). We then get the average accuracy for \(B_5\), see \(\mathbf{(\;2\;)}\) and lastly the average estimated probability for \(B_5\), see \(\mathbf{(\;3\;)}\). Repeat this for all bins and in our small example of \(9\) samples we end up with an ECE of \(0.10445\). A perfectly calibrated model would have an ECE of 0.
The images of binning above provide a visual guide of how ECE could result in very different values if we used more bins or perhaps binned the same number of items instead of using equal bin widths. Such and more drawbacks of ECE have been highlighted by several works early on
A model which minimises ECE, does not necessarily have a high accuracy
One of the most frequently mentioned issues with ECE is its sensitivity to the change in binning
We can see that bin 8 and 9 each contain only a single sample and also that half the bins now contain no samples. The above is only a toy example, however since modern models tend to have higher confidence values samples often end up in the last few bins
To mitigate these issues of fixed bin widths some authors
Binning-based evaluation with bins containing an equal number of samples are shown to have lower bias than a fixed binning approach such as ECE
Another frequently mentioned drawback of ECE is that it only considers the maximum estimated probabilities
Let’s say we trained two different models and now both need to determine if the same input image contains a person, an animal or no creature. The two models output vectors with slightly different estimated probabilities, but both have the same maximum confidence for “no creature”. Since ECE only looks at these top values it would consider these two outputs to be the same. Yet, when we think of real-world applications we might want our self-driving car to act differently in one situation over the other
A model is considered multi-class calibrated if, for any prediction vector \(q=(q_1,...,q_K) \in \Delta^K\), the class proportions among all values of \(X\) for which a model outputs the same prediction \(\hat{p}(X)=q\) match the values in the prediction vector \(q\).
\[\mathbb{P} (Y = k \; | \; \hat{p}(X)=q ) = q_k \;\;\;\:\:\:\: \forall k \in \{1,...,K\}, \; \forall q \in \Delta^K\]What does this mean in simple terms? Instead of \(c\) we now calibrate against a vector \(q\), with K classes. Let’s look at an example below:
On the left we have the space of all possible prediction vectors. Let’s zoom into one such vector that our model predicted and say the model has 10 instances for which it predicted the vector \(q\)=[0.1,0.2,0.7]. Now in order for it to be multi-class calibrated, the distribution of the true (actual) class needs to match the prediction vector \(q\). The image above shows a calibrated example with [0.1,0.2,0.7] and a not calibrated case with [0.1,0.5,0.4].
A model is considered class-wise calibrated if, for each class k, all inputs that share an estimated probability \(\hat{p}_k(X)\) align with the true frequency of class k when considered on its own:
\[\mathbb{P} (Y = k \; | \; \hat{p}_k(X)= q_k ) = q_k \;\;\;\;\;\; \forall k \in \{1,...,K\}\]Class-wise calibration is a weaker definition than multi-class calibration as it considers each class probability in isolation rather than needing the full vector to align. The image below illustrates this by zooming into a probability estimate for class 1 specifically: \(q_1=0.1\). Yet again, we assume we have 10 instances for which the model predicted a probability estimate of 0.1 for class 1. We then look at the true class frequency amongst all classes with \(q_1=0.1\). If the empirical frequency matches \(q_1\) it is calibrated.
To evaluate such different notions of calibration, some updates are made to ECE to calculate a class-wise error. One idea is to calculate the ECE for each class and then take the average
All the approaches mentioned above share a key assumption: ground-truth labels are available. Within this gold-standard mindset a prediction is either true or false. However, annotators might unresolvably and justifiably disagree on the real label
We have the same image as in our entry example and can see that the chosen label differs between annotators. A common approach to resolving such issues in the labelling process is to use some form of aggregation
In the same example as above, instead of aggregating the annotator votes we could simply use their frequencies to create a distribution \(P_{vote}\) over the labels instead, which is then our new \(y_i\). This shift towards training models on collective annotator views, rather than relying on a single source-of-truth motivates another definition of calibration: calibrating the model against human uncertainty
A model is considered human-uncertainty calibrated if, for each specific sample \(x\), the predicted probability for each class k matches the ‘actual’ probability \(P_{vote}\) of that class being correct.
\[\mathbb{P}_{vote} (Y = k \; | \; X = x ) = \hat{p}_k(x) \;\;\;\;\; \forall k \in \{1,...,K\}\]This interpretation of calibration aligns the model’s prediction with human uncertainty, which means each prediction made by the model is individually reliable and matches human-level uncertainty for that instance. Let’s have a look at an example below:
We have our sample data (left) and zoom into a single sample \(x\) with index \(i=1\). The model’s predicted probability vector for this sample is [0.1,0.2,0.7]. If the human labelled distribution \(y_i\) matches this predicted vector then this sample is considered calibrated.
This definition of calibration is more granular and strict than the previous ones as it applies directly at the level of individual predictions rather than being averaged or assessed over a set of samples. It also relies heavily on having an accurate estimate of the human judgement distribution, which requires a large number of annotations per item. Datasets with such properties of annotations are gradually becoming more available
To evaluate human uncertainty calibration three new measures are introduced in
where \(H(.)\) signifies entropy.
EntCE aims to capture the agreement between the model’s uncertainty \(H(\hat{p}_i)\) and the human uncertainty \(H(y_i)\) for a sample \(i\). However, entropy is invariant to the permutations of the probability values; in other words it doesn’t change when you rearrange the probability values. This is visualised in the image below:
On the left, we can see the human label distribution \(y_i\) , on the right are two different model predictions for that same sample. All three distributions would have the same entropy, so comparing them would result in 0 EntCE. While this is not ideal for comparing distributions, entropy is still helpful in assessing the noise level of label distributions.
\(RankCS = \frac{1}{N} \sum_{n=1}^{N} \mathbf{1} (argsort(y_i) = argsort(\hat{p}_i)),\)
where argsort simply returns the indices that would sort an array.
So, RankCS checks if the sorted order of estimated probabilities \(\hat{p}_i\) matches the sorted order of \(y_i\) for each sample. If they match for a particular sample \(i\) one can count it as 1; if not, it can be counted as 0, which is then used to average over all samples N.
Since this approach uses ranking it doesn’t care about the actual size of the probability values. The two predictions below, while not the same in class probabilies would have the same ranking. This is helpful in assessing the overall ranking capability of models and looks beyond just the maximum confidence. At the same time though, it doesn’t fully capture human uncertainty calibration as it ignores the actual probability values.
DistCE has been proposed as an additional evaluation for this notion of calibration. It simply uses the total variation distance \((TVD)\) between the two distributions, which aims to reflect how much they diverge from one another. DistCE and EntCE capture instance level information. So to get a feeling for the full dataset one can simply take the average expected value over the absolute value of each measure: \(E[\mid DistCE \mid]\) and \(E[\mid EntCE \mid]\). Perhaps future efforts will introduce further measures that combine the benefits of ranking and noise estimation for this notion of calibration.
We have run through the most common definition of calibration, the shortcomings of ECE and additional notions of calibration: multi-class, class-wise & human-uncertainty calibration. We also touched on some of the newly proposed evaluation measures and their shortcomings. Despite several works arguing against the use of ECE for evaluating calibration, it remains widely used. The aim of this blogpost is to draw attention to these works and their alternative approaches. Determining which notion of calibration best fits a specific context and how to evaluate it should avoid misleading results. Maybe, however, ECE is simply so easy, intuitive and just good enough for most applications that it is here to stay?
PLACEHOLDER FOR ACADEMIC ATTRIBUTION
BibTeX citation
PLACEHOLDER FOR BIBTEX