This tool measures agreement between coders beyond chance. Two coders can agree often simply because one category dominates: if 90 percent of segments are “neutral,” two careless coders agree 81 percent of the time by guessing neutral every time. Kappa corrects for that by comparing observed agreement with the agreement expected by chance given each coder’s marginal distributions. Cohen’s kappa handles two raters; Fleiss’ kappa generalizes to three or more. The tool reports kappa and a plain-language band from poor to almost perfect, following the widely used Landis and Koch guide.
How to use it
- Choose Cohen (2 raters) or Fleiss (3+ raters).
- For Cohen, paste both raters’ code lists in the same order.
- For Fleiss, paste one row per subject with counts per category.
- Click Calculate kappa.
- Read kappa and its band, then discuss disagreements.
Worked example
Two coders label six segments. Rater A says yes, yes, no, no, yes, no; Rater B says yes, no, yes, no, yes, no. Raw agreement is 4 of 6, or 67 percent — respectable until chance is considered. Both raters say yes half the time, so chance agreement is 50 percent, and kappa comes out around 0.33, only fair agreement. The pair then reviews the two disagreements, discovers one code definition is ambiguous, rewrites it, and recodes: agreement rises and the codebook improves. Kappa did its real job here — it started a conversation about definitions, not just produced a number.
Honest limits
Kappa has known quirks. With very skewed categories, high raw agreement can still yield low kappa (the kappa paradox), so always report raw agreement alongside kappa. Kappa treats all disagreements as equal, which is wrong for ordered scales where “agree” versus “strongly agree” is a smaller miss than “agree” versus “disagree” — weighted kappa fits ordered data better. And kappa measures reliability, not validity: two coders can agree perfectly on a wrong interpretation. Report kappa, raw agreement, the number of items double-coded, and how disagreements were resolved.
Who it is for
This tool is for qualitative researchers double-coding a sample, students learning reliability, and teams deciding whether a codebook is ready for full coding. A common rule: double-code at least 10–20 percent of the material, reach substantial agreement (kappa above 0.6), and only then code the rest independently.
Going further
Do not chase a high kappa by collapsing categories until everything agrees — merging distinct codes to inflate agreement hides real distinctions and weakens the analysis. Instead, treat persistent disagreement as data: log which items coders dispute, look for patterns in the disputes, and let those patterns refine the codebook or even generate a new theme. Report the kappa journey, not just the final value: initial kappa, what changed in the codebook, and final kappa tell readers the coding process was genuinely interrogated. If agreement stays low after two codebook revisions, the problem may be the material itself — ambiguous data, not careless coders — and that finding belongs in the limitations section, stated plainly. Report who the coders were and how they were trained, not just the kappa number. Two coders who built the codebook together will agree more than two independent coders applying it cold, and readers need that context to judge what the kappa means. State the unit of coding too — segment, sentence, or whole response — because agreement on long segments is easier than on exact boundaries, and a kappa without its unit is hard to compare with anyone else’s. Transparency about process turns a bare statistic into credible evidence. Double-code a fresh sample after any major codebook revision, rather than assuming the old kappa still holds: definitions change, and agreement must be re-earned, not inherited, after every change.
Related tools
Versión en español · Qualitative Coding Tag Hub · All free online tools.
Embed this tool on your website — free
Copy the code below to add this working tool to your own page. A small credit link stays with it — that’s all we ask.
Pingback: Thematic Node Network Explorer — Free Theme-Node Graph Builder