Inter-rater reliability calculator: Cohen’s kappa and ICC

Name: Cohen’s kappa and ICC calculator
Author: Richard Injante

Measure agreement between raters with Cohen’s kappa (unweighted and weighted) or the intraclass correlation coefficient, with confidence intervals.

Content reviewed by Richard Injante Published by SolucionesAhora.com Updated on 2026-10-08 How we verify

Calculator inputs

Enter your figures, or press “See example” to watch it work.
If your case is unusual, open “Advanced options”.
Read the result and, below it, what it means and how it was worked out.

Type of measure

Cohen’s kappa is for when the two raters classify into categories (healthy or ill, grade 1 to 4…). The intraclass correlation coefficient (ICC) is for when they measure a numeric quantity (a score, a measurement in millimetres…), with two or more raters.

Agreement table (counts)

One row per category of rater A and one column per category of rater B, with the number of cases for each combination. The table must be square. You can put the category names on the first line. 2×2 example: “20 5” and “10 15” means both say “yes” in 20 cases, A says “yes” and B “no” in 5, and so on.

Rows: rater A · columns: rater B

Measurements (subjects × raters)

Each row is a subject (or case) and each column a rater; everyone must have rated everyone. You can put the raters’ names on the first line. It takes from 2 subjects and 2 raters, up to 2,000 rows.

One row per subject, one column per rater

ICC model

It depends on how the raters were chosen. “Random raters, absolute agreement” (ICC 2) is the usual choice if you want to generalise to other raters and count the biases between them. “Fixed raters, consistency” (ICC 3) ignores systematic differences between raters. “One-way” (ICC 1) is for when each subject was rated by a different set of raters.

Advanced options

Weighting (optional)

With ordered categories (grade 1 to 4), missing by one category is not the same as missing by three. Weighted kappa gives partial credit to near disagreements: linear (proportional to the distance) or quadratic (penalises large ones more; equivalent to the ICC). For unordered categories, leave “unweighted”.

Unit of measurement (optional)

“Single measure” is the reliability of one individual rater (the usual choice to decide whether you can trust a single measurement). “Average of the raters” is the reliability of the mean of all of them, which is always higher.

Confidence level (optional)

The intervals are computed at this level. 95% is the standard.

See the result

Results

Cohen’s kappa0.400

Kappa CI: lower0.151

Kappa CI: upper0.649

Level of agreementfair

Observed agreement (%)70

Chance-expected agreement (%)50

Kappa standard error0.1270

PABAK (prevalence- and bias-adjusted kappa)0.400

Maximum possible kappa given those totals0.800

p valuep = 0.004

Totals by category
Category	Rater A (total)	Rater B (total)	Agree
1	25	30	20
2	25	20	15

See the working step by step

n = 50; categories = 2; standard error = 0.1270; p = 0.004 (test of κ = 0)
Maximum possible kappa given those totals = 0.800

In short

What it calculates: Measure agreement between raters with Cohen’s kappa (unweighted and weighted) or the intraclass correlation coefficient, with confidence intervals.
Formula used: Po = Σ w_ij p_ij Pe = Σ w_ij p_i· p_·j κ = (Po − Pe) / (1 − Pe)
Example: Example 1. Kappa for two raters.

Cohen’s kappa and ICC calculator

When two people (or two methods) classify or measure the same thing, you need to know how far they agree beyond chance. Percent agreement is not enough: with two categories, even raters answering by flipping a coin would agree half the time.

For categories (healthy or ill, grades 1 to 4) use Cohen’s kappa: it corrects agreement for chance and, if the categories are ordered, allows weighting. For numeric measurements, with two or more raters, use the intraclass correlation coefficient (ICC), in the form that fits your design. The calculator gives confidence intervals and the level of agreement against the usual benchmarks. Everything runs in your browser.

How it works

Kappa compares observed agreement (Po) with the agreement expected by chance (Pe) from each rater’s totals: κ = (Po − Pe) / (1 − Pe). It is 1 for perfect agreement, 0 if agreement is only what chance gives, and negative if it is worse. In the weighted version, disagreements between nearby categories count less.

The ICC divides between-subject variability by total variability, computed with an analysis of variance. There are six forms depending on whether the same raters rate every subject, whether they are treated as a sample or as fixed, whether absolute agreement or only consistency is required, and whether you report the reliability of one rater or of their average.

Formula

Po = Σ w_ij p_ij Pe = Σ w_ij p_i· p_·j κ = (Po − Pe) / (1 − Pe)
w_ij = 1 if i = j and 0 otherwise (unweighted); 1 − |i − j|/(k − 1) (linear); 1 − ((i − j)/(k − 1))² (quadratic)
Standard error (Fleiss, Cohen & Everitt, 1969): var(κ) = [Σ p_ij (w_ij − (w̄_i· + w̄_·j)(1 − κ))² − (κ − Pe(1 − κ))²] / (n (1 − Pe)²)
CI = κ ± z × SE PABAK = (k·Po − 1) / (k − 1)
Benchmarks (Landis & Koch): < 0 worse than chance · 0–0.20 slight · 0.21–0.40 fair · 0.41–0.60 moderate · 0.61–0.80 substantial · > 0.80 almost perfect
ICC(1,1) = (MSR − MSW) / (MSR + (k−1)·MSW)
ICC(2,1) = (MSR − MSE) / (MSR + (k−1)·MSE + k(MSC − MSE)/n)
ICC(3,1) = (MSR − MSE) / (MSR + (k−1)·MSE) the “k” (average) forms replace the denominator with MSR (or MSR + (MSC − MSE)/n for ICC 2)
ICC benchmarks (Koo & Li): < 0.50 poor · 0.50–0.75 moderate · 0.75–0.90 good · > 0.90 excellent

Worked example

Example 1. Kappa for two raters.
Two doctors classify 50 X-rays as “normal” or “abnormal”. They agree on 20 “normal” and 15 “abnormal” and disagree on 15. Observed agreement is 70% and chance would give 50%, so κ = 0.40 (standard error 0.127; 95% CI 0.151 to 0.649): fair agreement, not good, even though 70% may sound high.

Example 2. The kappa paradox.
In 100 cases two raters agree on 90, but almost all are of the same type (85 agree on “yes”, 5 on “no”). Observed agreement is 90%, but chance alone would give 82%, so κ = 0.444, only moderate. When one category is very common, kappa drops even with high agreement; in that case also report percent agreement and PABAK (0.80).

Example 3. ICC for six subjects and four judges.
The classic Shrout and Fleiss (1979) data: six subjects rated by four judges. ICC(2,1) is 0.290 (95% CI 0.019 to 0.761), poor, because the judges differ a lot in their average level. ICC(3,1), which ignores those systematic differences, is 0.715 (0.343 to 0.946). Choosing the wrong form changes the conclusion, which is why you must justify it.

Explanation

Why percent agreement is not enough

With two categories, two raters answering at random would agree about half the time. Percent agreement mixes real agreement and the agreement that happens by chance. Kappa subtracts that part and measures how much agreement there is above chance.

Cohen’s kappa: how to read it

**1** is perfect agreement, **0** is chance agreement and negative values mean less agreement than chance.
Landis and Koch (1977) benchmarks: slight up to 0.20; fair up to 0.40; moderate up to 0.60; substantial up to 0.80; almost perfect above. They are a convention, not a strict criterion.
Look at the confidence interval, not just the value: with few cases it can be very wide.
**Weighted kappa**: for ordered categories. Quadratic weighting equals the ICC and is the most common.
**The kappa paradox**: when one category is very frequent, or raters use categories with very different frequencies, kappa is low even with high agreement. Compare it with percent agreement and PABAK.

ICC: choosing the right form

**ICC 1**: each subject was rated by a different set of raters.
**ICC 2**: the same raters rate everyone and are treated as a sample of possible raters; absolute agreement counts (it includes biases between them). The usual choice.
**ICC 3**: the raters are the only ones of interest (fixed); only consistency counts, that is, ranking subjects the same way even if one rater always scores higher.
**Single or average measure**: the first to decide whether one rater is reliable; the second if in practice you will use the mean of all.
Koo and Li (2016) benchmarks: below 0.50 poor; up to 0.75 moderate; up to 0.90 good; above that, excellent. They apply to the confidence interval, not just the value.

Common mistakes

Reporting only percent agreement.
Using unweighted kappa with ordered categories instead of weighted.
Choosing the ICC form without justifying it (it changes the result).
Treating Landis and Koch’s benchmarks as an absolute criterion.
Comparing kappas across studies with very different prevalences.

Limits of this calculator

Kappa here is for two raters; for three or more you need Fleiss’ kappa, which is not included.
The ICC requires every rater to rate every subject; missing data are not allowed.
The kappa interval is the asymptotic one of Fleiss, Cohen and Everitt; it is approximate for tables with few cases.
ICC intervals assume roughly normal measurements.
It does not include the Bland-Altman plot, useful for seeing bias between two methods.

Notice

This is an estimate. For publication, check the figures with your statistics package and justify your choice of model.

Frequently asked questions

What is Cohen’s kappa?

A measure of agreement between two raters on categories that discounts the agreement expected by chance. It ranges from −1 to 1; 1 is perfect agreement and 0 is chance-level agreement.

What kappa counts as good?

On Landis and Koch’s benchmarks, 0.61 to 0.80 is substantial and above 0.80 almost perfect. It is a convention: some clinical fields demand more.

When should I use weighted kappa?

When the categories are ordered (grade 1 to 4): a disagreement between adjacent grades weighs less than one between extremes. With unordered categories, use the unweighted one.

Why is my kappa low if we nearly always agree?

That is the kappa paradox: if almost all cases fall into one category, chance already explains most of the agreement. Also report percent agreement and PABAK.

Which ICC form should I choose?

The most common is two-way random effects with absolute agreement (ICC 2). If the raters are fixed and only the ranking of subjects matters, use consistency (ICC 3). Justify it in your report.

What is the difference between single-measure and average ICC?

Single measure is the reliability of one rater alone; average is that of the mean of all of them, always higher. Report the one that matches how you will use the measurement.

What ICC counts as excellent?

Koo and Li suggest: below 0.50 poor, 0.50 to 0.75 moderate, 0.75 to 0.90 good and above 0.90 excellent, applying the benchmarks to the confidence interval.

Is my data sent to a server?

No. The calculation runs in your browser.

Sources and references

The formula behind this calculator is not ours. These are the sources it comes from, in APA 7 format, in case you want to go to the original or cite them in your own work.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104 https://doi.org/10.1177/001316446002000104 The definition of Cohen’s kappa.
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969). Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72(5), 323–327. https://doi.org/10.1037/h0028106 https://doi.org/10.1037/h0028106 The standard error of kappa, simple and weighted.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310 https://doi.org/10.2307/2529310 The agreement levels (slight, fair, moderate, substantial, almost perfect).
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420 https://doi.org/10.1037/0033-2909.86.2.420 The three ICC forms and their intervals; the example data come from this paper.
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012 https://doi.org/10.1016/j.jcm.2016.02.012 The guide to choosing the ICC model and the poor, moderate, good and excellent levels.

Need to calculate something else?

These tools are often used alongside this calculator.