Inter-Rater Reliability PublicationsComputing Agreement Coefficients

 Home Book Excerpts Software Research Papers Store Contact Discussion Corner

 AgreeStat for Excel AgreeStat for SAS R functions for R users

AgreeStat for Excel (Windows & Mac)

A Software for Analyzing the Extent of Agreement Among Raters with MS Excel

 Chance-Corrected Agreement Coefficients Intraclass Correlation Coefficients

AgreeStat is a self-automated workbook that contains a VBA (Visual Basic for Applications) program, and therefore requires no installation. It is a point-and-click program that is easy to use, and that computes many well-known inter-rater reliability coefficients along with their standard errors. To use AgreeStat, choose one of the following 2 options:

 Download a trial version of AgreeStat 2015.6.1-Windows (It contains sample datasets with instructions for analysis). If you get an error message when opening the workbook, check here. AgreeStat user's guide may be downloaded here. Download the full version of AgreeStat 2015.6.1-Windows for \$49.95 Note: AgreeStat-Windows has been successfully tested with both Excel 2007, 2010, and 2013. It may not work well with Excel 2003 and earlier versions.

 Download a trial version of AgreeStat 2015.6-Mac, with User's Guide, and Sample Datasets for Testing (The sample datasets also contain instructions for analysis). If you get the error message "Error in loading DLL," then check here. Download the full version of AgreeStat 2015.6-Mac now for \$49.95 Note: AgreeStat Mac will NOT work with Excel 2008, which does not support Visual Basic for Application (VBA). You will need Excel 2011 (SP1) or 2016 (version 15.27 or higher) for Mac to use AgreeStat. Please test the trial version on your system before buying.

• Using Comma as the Decimal Symbol? Check here!
• AgreeStat
This is your ideal solution if your data contains missing values, or you want to compute weighted chance-corrected agreement coefficients with your own custom weights (as opposed to using standard linear or quadratic weights), or you want to compute intraclass correlation coefficients. With AgreeStat,

• You may compute Cohen's Kappa, Krippendorff's Alpha, Gwet's AC1, Fleiss' Kappa, and more ...
• You may compute Weighted agreement coefficients using a wide variety of weights, including Linear weights, Quadratic weights, Ordinal weights, Circular weights, and many more. Or specify your own Custom Weights as well. This feature is essential for researchers who want to implement their specific view of the nature of disagreements.
• You may compute Intraclass Correlation Coefficients with missing values.
• You will obtain inter-rater reliability coefficients along with their standard errrors, confidence intervals, and p-values.

Only a Few Steps from Data Entry to Finished Report

• From the screen shown below, choose the "2 Raters" or the "3 Raters or More" tab, depending on the number of observers involved in your study. Then specify the format in which you want to enter your data.

• Choose Your Method of Analysis and Select Your Data

• Specify the options you want for your analysis (Chance-corrected Agreement Coefficients):

• Enter a 2x2 Contingency Table

• Define the Intraclass Correlation Model

• Click on the Execute button, and get the results on the output worksheet

• When dealing with missing ratings (i.e. some observers do not score all subjects) then you must select the option "2 Columns of Raw Data" if dealing with 2 raters.
• If you want to capture your own custoim weights, you will use the following screen:

• Your can enter your ratings for 3 raters or more using the screen shown below. You will have the opportunity to select the workbook containing your data, as well as a specific worksheet.

• Getting an Error Message when Launching AgreeStat / Windows?
• This problem exists mainly on computers on which a security update recently released by Microsoft was installed. Please check the link http://support.microsoft.com/kb/3025036/EN-US
to see what Microsoft suggests to do to get the program running.
• Back to the top

• AgreeStat / Mac: "Error in loading DLL" when Laucnhing the program
• You may first want to test if VBA (Visual Basic for Applications) is installed and working. Click Tools-Macros-Record New Macro. If you get a message, "Cannot Record Macro", it means VBA isn't working and AgreeStat won't either. In this case, proceed to the next step.
• The message "Error in loading DLL" is often an indication that it is necessary to reinstall Microsoft Excel and to make sure VBA is Enabled. The following link http://support.microsoft.com/kb/2435801 may help.

Back to the top

• Using Comma as the Decimal Symbol?
If you are from a country that uses the comma as the decimal symbol, and/or the dot as the list separator, you need to take the following actions in order to run these Excel macros smoothly
:
• Just for the purpose of running the program, change the decimal symbol from comma (,) to dot (.), and the list separator from whatever it is to comma (,) ? This is accomplished as follows:

1) Go to CONTROL PANEL
2) Clock, Language, and Region
3) Region and Language
4) Additional settings ...
5) Then you can change the decimal symbol, and the list separator.