Agreement among 4 Raters: Unweighted Analysis of the
Distribution of Raters by Subject, Group and Category

Input Data

The dataset shown below is included in one of the example worksheets of AgreeStat 360, and can be downloaded from that program. It shows the distribution of 4 raters by unit (or subject), unit group, and category. That is, 4 raters assigned each of the 15 units to one of the 3 categories a, b, and c, and the ratings were organized in the form a distribution of raters.

AgreeStat360 can process the ratings data in this format as well, separately for each group and for all groups combined.

Analysis with AgreeStat/360

To see how AgreeStat360 processes this dataset to generate various agreement coefficients, please play the video below.  This video can also be watched on youtube.com for more clarity if needed.

Results

• The output that AgreeStat360 produces is shown below.  Each of the subject groups A and B was analyzed separately before the were combined for an overall abalysis.  Each group processed contains 2 parts:

  • Summary data:
    The first part of this output shows some general summary statistics.

  • Unweighted analysis:
    A total of 5 agreement coefficients, among which are Fleiss' kappa, Gwet's AC1, and more. Each agreement coefficient is associated with a standard error, the 95% confidence interval and the p-value