logoInter-Rater Reliability Publications
Inter-Rater Reliability With R

Home Book Excerpts Software Research Papers Store Contact Discussion Corner

R functions for calculating agreement coefficients

On this page R users may download a number of functions that can be used to compute various chance-corrected agreement coefficients, their standard errors, confidence intervals, and p-values. Another script file named paired t-test for agreement coefficients.r contain various functions that can be used to test the difference between two correlated agreement coefficients for statistical significance. The agreement coefficients considered for the paired t-test include Fleiss' Kappa, Gwet's AC2, Conger's Kappa, Krippendorff's Alpha and Brennan-Prediger Coefficient, and the approach used is based on the Linearization method of Gwet, K. L. (2016). Testing the Difference of Correlated Agreement Coefficients for Statistical Significance, Educational and Psychological Measurement, Vol 76(4) 609-637.

The functions used for computing agreement coefficients are organized in three R script files, corresponding to the three ways your ratings must be organized. An additional script file contain functions that could be used to create various weights for perfoming the weighted analysis. These script files are the following (click the filename to download the script file):

agree.coeff2.r (for 2 raters only with ratings in a contingency table)

agree.coeff3.dist.r (for 2 raters or more)

agree.coeff3.raw.r (for 2 raters or more - requires the script file "weights.gen.r" to be loaded as well.)

weights.gen.r (this script is required to run the functions in the script file agree.coeff3.raw.r.)

paired t-test for agreement coefficients.r This script file contains functions that implement the paired t-test for testing the difference between 2 agreement coefficients for statistical significance, based on the linearization method of Gwet (2016)

In order to use any of the functions contained in these script files, you need to read the appropriate script into R. If you want to use the functions contained in"agree.coeff2.r" for example, then you will read this file into R as follows: