Cohen's Kappa and Krippendorff's Alpha Calculator
Paste your coding spreadsheet and get the coefficient that fits your data, with its confidence interval and a table showing which categories the disagreement is concentrated in.
- Cohen's, weighted, and Fleiss' kappa, and Krippendorff's alpha
- Handles empty cells and any number of coders
- Calculated in your browser: nothing is uploaded
Your data
One row per coded unit, one column per coder. Copy the range from your spreadsheet and paste it here. Empty cells are treated as "not coded."
There's nothing to calculate yet.
Paste your table on the left and press Calculate. If you want to see how it works first, load one of the two examples.
The calculator picks the measure that fits your data and also shows the others, so you can report whichever one you are asked for.
Three questions decide the coefficient
How many coded, whether they coded everything, and whether the categories have an order. The calculator applies this rule on its own, but it helps to know why it chooses what it chooses.
| Your data | Main measure |
|---|---|
| Two coders, everyone coded everything, unordered categories | Cohen's kappa |
| Two coders, ordered categories | Weighted kappa, linear or quadratic |
| Three or more, everyone coded everything | Fleiss' kappa |
| Any number, with empty cells | Krippendorff's alpha |
Alpha works in all four cases, which is why it is always shown. The other three are special cases of it: when the data are complete, they give very similar values. The difference appears with empty cells, where kappa drops the whole unit and alpha keeps it, weighting it by how many coded it.
This does not measure agreement between you and an AI model
Why not, even if the number comes out the same
The three coefficients assume two independent coders. The correction for chance consists, precisely, of subtracting the agreement that would have occurred if each had decided on their own.
When an AI proposes a coding and you review it, that independence doesn't exist: you saw the proposal before deciding. The resulting number measures how much you accepted, not how much two separate judgments coincided. A high value can mean the model coded well, or that you reviewed with little attention, and the coefficient can't tell the two apart.
What you can report is the review rate: what proportion of what the model proposed you ended up changing. It's an honest figure, it's calculated without assumptions, and it describes exactly what happened.
And if you need a real coefficient, there is a way to get one: code a sample blind, without seeing the model's output, and compare only afterward. There the two judgments are independent and the measure means what it says it measures again.
The cutoffs are conventions, not statistical thresholds
The table almost everyone cites is Landis and Koch's, from 1977. It is used so widely that it's worth knowing, but its own authors didn't back it with evidence.
| Value | Landis and Koch label |
|---|---|
| Less than 0.00 | Worse than chance |
| 0.00 to 0.20 | Slight |
| 0.21 to 0.40 | Fair |
| 0.41 to 0.60 | Moderate |
| 0.61 to 0.80 | Substantial |
| 0.81 to 1.00 | Almost perfect |
Two caveats before using it. First: the value depends on how many categories your codebook has, and with few categories the same work gives a higher number, so two studies with different numbers of codes aren't comparable by their label. Second: for content analysis, Krippendorff proposes a more demanding and better-argued criterion, 0.80 for drawing conclusions and 0.667 as a floor for results presented as tentative. Always report the coefficient with its interval and the number of units.
It depends on three things: how many coders there are, whether all of them coded all the units, and whether the categories have an order. With two coders, complete data, and unordered categories, Cohen's kappa. With three or more and complete data, Fleiss' kappa. If there are empty cells, Krippendorff's alpha, which is the only one of the three that doesn't drop those units. If the categories are ordered, the weighted version of kappa or ordinal alpha, because confusing "quite a bit" with "a lot" is not the same as confusing "quite a bit" with "none." The calculator applies this rule on its own and still shows the other measures.
No, and it's worth saying clearly because it's the most frequent question. These coefficients assume two independent coders. If you saw the model's proposal first and decided afterward, you are no longer independent of it: the resulting number measures how much you accepted, not how much two separate judgments coincided. What you can report is the review rate, meaning what proportion of what was proposed you changed. If you really want a coefficient, there is a way to get one: code a sample blind, without seeing the model's output, and only then compare. There the independence exists and the measure means something again.
Yes, with Krippendorff's alpha. Cohen's kappa and Fleiss' kappa need all coders to have coded all units, so they drop the incomplete units. Alpha keeps them: it weights each unit by how many coded it, and only leaves out those with a single coding, because with one value there is nothing to compare. The calculator tells you how many units each measure dropped.
There is no cutoff with a statistical basis. The table most people cite is Landis and Koch's from 1977, which calls the 0.61 to 0.80 range "substantial," but its authors didn't back it with evidence: it's a convention. Krippendorff proposes a more demanding and better-argued criterion, 0.80 to conclude and 0.667 as a floor for tentative results. Also, the value depends on how many categories you use: with few categories the same work gives a higher number. It's best to report the coefficient with its interval and the number of units, and not rely only on the label.
No. All the calculation happens in your browser, on your computer. The page doesn't send what you paste anywhere, doesn't store anything, and works offline once loaded. You can check by opening the network tab of your browser's developer tools while you calculate: there are no requests.
Both are defensible and give very similar values. They differ in how they estimate the agreement expected by chance: Cohen's kappa uses each coder's marginals separately, so it tolerates one coder using a category much more than the other; alpha uses a common distribution and corrects for sample size. If you are going to compare against previous studies, report the coefficient they used. If you have to choose just one for all your work, alpha applies unchanged when you later add a third coder or empty cells appear.
Because kappa has a closed formula for its standard error, published by Fleiss, Cohen, and Everitt in 1969, and alpha doesn't have an equivalent that is reliable. Krippendorff himself recommends estimating the interval by resampling. Here it's done by resampling the units a thousand times and taking the 2.5th and 97.5th percentiles, which is what the usual implementations do. That's why alpha's interval can change by a tiny amount between two calculations with the same data.
Agreement is measured on what you coded
HILO gathers the passages of each topic across your documents, with the page of each one, so you can code and edit them together.