Estimators of different delta coefficients based on the unbiased estimator of the expected proportions of agreements

To measure the degree of agreement between two observers that independently classify $n$ subjects within $K$ categories, it is common to use different kappa type coefficients, the most common of which is the ${\kappa }_C$ coefficient (Cohen’s kappa). As ${\kappa }_C$ has some weaknesses -such as its poor performance with highly unbalanced marginal distributions-, the $\mathrm{\Delta }$ coefficient is sometimes used, based on the $delta$ response model. This model allows us to obtain other parameters like: (a) the ${\alpha }_i$ contribution of each $i$ category to the value of the global agreement $\mathrm{\Delta }=\sum {\alpha }_i$; and (b) the consistency ${\mathcal{S}}_i$ in the category $i$ (degree of agreement in the category $i$), a more appropriate parameter than the kappa value obtained by collapsing the data into the category $i$. It has recently been shown that the classic estimator ${\widehat{\kappa }}_C$ underestimates ${\kappa }_C$, having obtained a new estimator ${\widehat{\kappa }}_{CU}$ which is less biased. This article demonstrates that something similar happens to the known estimators $\widehat{\mathrm{\Delta }}$, ${\widehat{\alpha }}_i$, and ${\widehat{\mathcal{S}}}_i$ of $\mathrm{\Delta }$, ${\alpha }_i$ and ${\mathcal{S}}_i$ (respectively), proposes new and less biased estimators ${\widehat{\mathrm{\Delta }}}_U$, ${\widehat{\alpha }}_{iU}$, and ${\widehat{\mathcal{S}}}_{iU}$, determines their variances, analyses the behaviour of all estimators, and concludes that the new estimators should be used when $n$ or $K$ are small (at least when $n\le 50$ or $K\le 3$). Additionally, the case where one of the raters is a gold standard is contemplated, in which situation two new parameters arise: the $conformity$ (the rater’s capability to recognize a subject in the category $i$) and the $predictivity$ (the reliability of a response $i$ by the rater).