Home / Blog
Kilem L. Gwet, Ph.D.
Inter-Rater Reliability Blog
Notes on the design and analysis of inter-rater reliability studies — the techniques, paradoxes, and open questions that don't always make it into a textbook chapter.
About this page. This blog is moving here from Blogger post by post. Posts marked "Read on this site" have been migrated in full; the rest still link out to the original Blogger post until they're brought over.
2023
On Bangdiwala's B Coefficient and Its Variance
A proposed variance estimator for Bangdiwala's B agreement statistic, checked against a 10,000-subject Monte Carlo simulation across a range of sample sizes.
Migrated · Jan 11, 2023
2021
Cohen's Kappa Paradoxes Make Sample Size Calculation Impossible
Why the well-known Kappa paradoxes have a second, less-discussed consequence: no amount of additional subjects can guarantee a small standard error for Kappa-family coefficients.
Migrated · Aug 28, 2021
2021
Agreement Among 3 Raters or More When a Subject Can Be Rated by No More Than 2 Raters
A partially-crossed design for measuring multi-rater agreement when cost or burden rules out having every rater score every subject.
Migrated · Mar 30, 2021
2021
Testing the Difference Between 2 Agreement Coefficients for Statistical Significance
How to test whether two agreement coefficients differ significantly, covering both the correlated and uncorrelated cases, with references to studies that have used the technique.
On Blogger · Feb 22, 2021
2021
New Peer-Reviewed Article
Announces a published article clarifying that the Fleiss generalized kappa variance formula built into SPSS and some R packages is only valid for hypothesis testing, not for confidence intervals.
On Blogger · Feb 16, 2021
2020
Large-Sample Variance of Fleiss Generalized Kappa Coefficient
A detailed walkthrough of why SPSS and certain R packages compute a variance for Fleiss' kappa that isn't appropriate for general use, and a linearization-based alternative.
On Blogger · Jul 20, 2020
2019
Handbook of Inter-Rater Reliability, 5th Edition
Announces work in progress on the 5th edition, split into two volumes, with early chapter drafts opened up for reader feedback before publication.
On Blogger · Nov 8, 2019
Looking for the complete archive back to 2013? The full history is still available on Blogger.