Handing two people the same codebook and expecting them to independently land on consistent results is a reasonable hope and a poor plan on its own. A codebook, however well-written, still leaves real room for individual judgment - which is exactly what a second coder is there to check in the first place. Getting a coding team to genuinely agree takes something more deliberate than a shared document: a calibration process before real coding starts, a way to notice drift once it's underway, and an actual, agreed-upon process for resolving the disagreements that will still happen even after both of those are in place.
Table of Contents¶
- Why a Shared Codebook Alone Isn't Enough
- Calibration: A Pilot Round Before It Counts
- Catching Drift Once Coding Is Underway
- A Real Process for Resolving Disagreements
- When to Stop Calibrating and Just Measure Agreement
- A Worked Example
- FAQ
Why a Shared Codebook Alone Isn't Enough¶
Even a well-written codebook, covered in depth in our guide to building one, can't anticipate every real case that shows up once actual coding starts - two coders reading the same careful definitions will still occasionally draw the boundary in different places on a genuinely ambiguous response, simply because written language can only narrow interpretation so far before it needs to be tested against real, messy data. This isn't a sign of a bad codebook; it's an expected, normal part of the process, and treating a first-draft codebook as though it should produce perfect agreement on its own sets a team up to be surprised, and possibly discouraged, by disagreements that a deliberate calibration process would have caught and resolved calmly, before they mattered.
Calibration: A Pilot Round Before It Counts¶
The single most effective step for getting a coding team aligned is a calibration round: both coders independently code the same small batch of real data - fifteen to twenty-five responses is usually enough - before either of them starts on the actual, full dataset. The results get compared directly, response by response, not just as a summary statistic. Every disagreement becomes a specific, concrete discussion: what did each coder see in this response that led them to different codes, and which reading (or possibly a codebook revision) resolves it most defensibly. This calibration round typically surfaces exactly the kind of overlapping or vaguely-defined codes covered in our guide on what makes a good qualitative codebook - and finding them here, on a small pilot batch, is far cheaper than finding them for the first time three hundred responses into full coding.
Catching Drift Once Coding Is Underway¶
Agreement achieved during calibration doesn't automatically hold for the rest of a long coding project - coders can drift apart gradually as fatigue sets in, as memory of the calibration discussion fades, or as new kinds of ambiguous cases appear that the original pilot batch happened not to include. Building in periodic check-ins - re-comparing a small, fresh sample of independently coded responses partway through the full dataset, not just at the very start - catches this drift while there's still time to correct it, rather than discovering after the fact that the first half of coding used a subtly different standard than the second half. A simple, practical cadence: a short calibration check after every quarter of the dataset has been coded, comparing a handful of recent responses each time, is usually enough to catch meaningful drift without turning the coding process itself into a series of constant interruptions.
A Real Process for Resolving Disagreements¶
Even a well-calibrated team with a strong codebook will hit occasional real disagreements once the full dataset is underway, and having an actual, agreed-upon process for resolving them - decided in advance, not improvised in the moment - keeps individual disagreements from becoming inconsistent, ad hoc calls made differently each time. A few common, workable approaches: the two coders discuss the specific case directly and reach a joint decision, which gets logged (the same memoing practice covered in our guide on analytic memoing) so the same reasoning applies the next time a similar case appears; a third, more senior team member acts as a tiebreaker for cases the two original coders can't resolve between themselves; or, for a project with enough volume, a running log of unresolved disagreements gets reviewed together periodically, in a batch, rather than resolved one at a time under time pressure in the middle of active coding.
When to Stop Calibrating and Just Measure Agreement¶
Calibration is a training process; a formal inter-rater reliability check, covered from a human-AI comparison angle in our guide on running an inter-rater reliability check, is a measurement of how well that training actually worked. The two are related and sequential, not the same step: calibration continues, with active discussion and codebook revision, until the team is genuinely aligned on how to handle the range of cases that show up in real data. Once that alignment feels stable - agreement on new calibration samples is consistently high, with no new codebook revisions needed - it's time to stop actively calibrating and instead run a clean, independent reliability check on a fresh sample, without further discussion contaminating the result, to get an honest, reportable measure of how well the team actually agrees now that training is complete.
A Worked Example¶
A two-person research team preparing to code 400 open-ended employee survey responses starts with a calibration round on 20 responses, coded independently. Comparing results reveals six disagreements, four of which trace back to the same underlying issue: both coders are drawing the line between "Management Communication" and "Management Decision-Making" differently, since the codebook's original definitions left real room for either reading on responses that touched both. The team discusses each disagreement, agrees on an explicit tiebreaker rule ("code as Communication if the complaint is about how a decision was conveyed; code as Decision-Making if the complaint is about the decision's content"), and adds it to the codebook. A second calibration round on a fresh 20 responses shows only one disagreement, on a genuinely ambiguous edge case rather than a systematic pattern - a sign the team is ready to move into full coding. Midway through the full 400, a scheduled check-in comparing a fresh sample of 15 responses shows agreement holding steady, confirming no meaningful drift has crept in since the original calibration.
FAQ¶
How many responses should a calibration round use?
Fifteen to twenty-five is usually enough to surface real disagreements and codebook gaps without consuming so much of the dataset that little remains for the actual coding pass. The goal is catching patterns, not exhaustively testing every possible case.
Should disagreements during calibration be resolved by majority vote or discussion?
Discussion is stronger for a two-person team, since majority vote isn't meaningful with only two coders anyway - the point of calibration is understanding why each coder read a case differently, which a vote alone doesn't surface.
How often should drift checks happen during a long coding project?
A reasonable default is after each quarter of the dataset is coded, though shorter or higher-stakes projects may warrant checking more frequently. The right cadence balances catching drift early against not interrupting the coding process too often.
What if the two coders can't agree even after discussion?
Bring in a third person as a tiebreaker if one is available, or explicitly flag the case as genuinely ambiguous and document both readings rather than forcing an artificial resolution - some cases are legitimately unresolvable from the text alone, and pretending otherwise is less honest than acknowledging it.
For the statistical side of this process, see AI Coding vs. Human Coding: Running Your Own Inter-Rater Reliability Check and What Makes a Good Qualitative Codebook.