Home |
Book Excerpts |
Software |
Research Papers |
Store |
Contact |
Discussion Corner |

# R functions for calculating agreement coefficients

On this page R users may download a number of functions that can be used to compute various chance-corrected agreement coefficients, their standard errors, confidence intervals, and p-values. Another script file named paired t-test for agreement coefficients.r contain various functions that can be used to test the difference between two correlated agreement coefficients for statistical significance. The agreement coefficients considered for the paired t-test include Fleiss' Kappa, Gwet's AC2, Conger's Kappa, Krippendorff's Alpha and Brennan-Prediger Coefficient, and the approach used is based on the Linearization method of* Gwet, K. L. (2016). Testing the Difference of Correlated Agreement Coefficients for Statistical Significance, Educational and Psychological Measurement, Vol 76(4) 609-637*.

The functions used for computing agreement coefficients are organized in three R script files, corresponding to the three ways your ratings must be organized. An additional script file contain functions that could be used to create various weights for perfoming the weighted analysis. These script files are the following (click the filename to download the script file):

**agree.coeff2.r** (for 2 raters only with ratings in a contingency table)

- The functions in this script file compute various agreement coefficients and their standard errors when dealing with 2 raters, and ratings that are organized in the form of a contingency table (or a square matrix) showing counts of subjects by rater and category. You may use this format if each rater rated all subjects. Otherwise subjects rated by one rater and not by the other may not be properly classified. More details regarding the use of these functions can be found in the readme_r_functions.pdf file.

** agree.coeff3.dist.r** (for 2 raters or more)

- The functions in this script file compute various agreement coefficients and their standard errors when dealing with multiple raters, and ratings that are organized in the form of an n × q table showing counts of raters by subject and category. Here n represents the number of subjects and q the number of categories. More details regarding the use of these functions can be found in the readme_r_functions.pdf file.

**agree.coeff3.raw.r** (for 2 raters or more - requires the script file "weights.gen.r" to be loaded as well.)

- The functions in this script file compute various agreement coefficients and their standard errors when dealing with multiple raters, and rating data organized in the form of an n × r table showing the (alphanumeric) raw ratings that the raters assigned to the subjects. Here n represents the number of subjects and r the number of raters. The data is presented in the form of n rows containing r ratings each. More details regarding the use of these functions can be found in the readme_r_functions.pdf file.

**weights.gen.r** (this script is required to run the functions in the script file agree.coeff3.raw.r.)

- The functions in this script file generate various weights to be used when computing

weighted agreement coefficients. More details regarding the use of these functions can be found in the readme_r_functions.pdf file.

**paired t-test for agreement coefficients.r** This script file contains functions that implement the paired t-test for testing the difference between 2 agreement coefficients for statistical significance, based on the linearization method of Gwet (2016)

In order to use any of the functions contained in these script files, you need to read the appropriate script into R. If you want to use the functions contained in"agree.coeff2.r" for example, then you will read this file into R as follows:

>source("C:\\YourDirectory\\agree.coeff2.r")