How to tell if two coders agree: Cohen’s kappa in 5 steps

Pick a sample of the material, for example 20 responses or 10 to 20% of the units, and agree on the codebook first: each category with a short definition and an example. Each person assigns a single category per unit, without seeing what the other chose and without discussing doubtful cases until both are done.

Copy the codes into a spreadsheet with three columns: the unit number and the category each coder chose. Categories must be spelled exactly the same in both columns, capitalization included: “Cost” and “cost” count as two different categories. An empty cell is a unit that coder didn’t code.

Cross the two columns in a contingency table: rows are what coder 1 chose; columns, what coder 2 chose. The diagonal holds the matches.

In the example they agreed 6 times on “access”, 5 on “cost” and 5 on “treatment”: 16 out of 20. The observed agreement is p_o = 16/20 = 0.80.

If each coder had assigned categories at random, at the rate they used each one, they would still have matched sometimes. That expected agreement is calculated per category, multiplying coder 1’s proportion by coder 2’s, and adding them up. Coder 1 used the categories 8, 6 and 6 times; coder 2, 7, 7 and 6:

p_e = (8/20 × 7/20) + (6/20 × 7/20) + (6/20 × 6/20) = 0.140 + 0.105 + 0.090 = 0.335

Kappa compares what was observed with what was expected:

κ = (p_o − p_e) / (1 − p_e) = (0.80 − 0.335) / (1 − 0.335) = 0.699

A kappa of 1 is perfect agreement; 0, the agreement chance alone would produce; a negative value, less than chance.

Report kappa with its interval, the number of units and categories, and the observed agreement. The Landis and Koch scale calls a value between 0.61 and 0.80 “substantial”, but that is a convention, not a threshold.

Also look at where the disagreement concentrates: if one category gathers most of it, refine its definition and repeat the measurement with a new sample. With three or more coders, or with empty cells, Cohen’s kappa isn’t enough: use Fleiss’ kappa or Krippendorff’s alpha.