All guides  ›  Verification and method comparison

How do I assess agreement when results are categories?

Compare the paired categories in a table and read the disagreements in both directions. An overall percentage can be useful, but the table shows whether one procedure is more likely to call a sample positive or negative.

Worked example at a glance

45 of 50

Overall agreement

18 of 20

Positive agreement

27 of 30

Negative agreement

Put the two procedures in one table

Suppose two procedures classify the same 50 fictional specimens as positive or negative. Method A is the comparator and method B is the procedure being evaluated. Each specimen appears once in the table. The rows show A’s calls and the columns show B’s calls. [1]

AB +B −
Positive182
Negative327
Paired results for 50 fictional specimens. The plus and minus signs indicate positive and negative calls. Method A defines the rows and the denominators.

Start with the cells where the procedures agree. Both call 18 specimens positive and 27 negative. That gives 45 agreements out of 50, or 90% overall agreement. Now read the other two cells. B calls two of A’s positive specimens negative and calls three of A’s negative specimens positive.

Read each direction of agreement

Positive percent agreement asks how often B also called a specimen positive among specimens A classified as positive. That is 18 out of 20, or 90%. Negative percent agreement asks how often B also called a specimen negative among specimens A classified as negative. That is 27 out of 30, also 90%. State the comparator and denominators so the reader can reproduce the interpretation. [1]

With method A as comparator, B agrees with 18 of 20 positive calls and 27 of 30 negative calls. The five disagreements split two and three by direction.
The denominator changes with the comparator category. In this example both percentages are 90%. Open the figure full size.

All three percentages happen to be 90% in this example. The counts show why they are still different summaries. In another dataset, the proportion of specimens in each category can make the overall result look reassuring while agreement is weaker for a less common category.

Try reading the first row in words before calculating a percentage. Method A called 20 specimens positive. B agreed on 18 and disagreed on two. Reading down the positive column answers a different question because the denominator would then be the 21 specimens B called positive. Reversing the comparator changes the calculation, even though the same 50 pairs appear in the table. [1]

Decide what the comparison supports

With an ordinary comparator, these are estimates of agreement relative to method A. A result can disagree with A for several reasons, including a difference near a threshold, variability in either procedure or a difference in how the methods define the category. An independent reference for the condition would be needed to interpret the results as sensitivity and specificity. [1]

If a numerical measurement is converted to a category, retain the numerical results where possible. The table tells you which pairs crossed the category boundary in opposite directions. The underlying numbers can show whether those pairs were close to the threshold. If the method also reports equivocal or invalid results, describe how those outcomes were handled and retain them in the study record. [1] [2]

Keep a specimen count alongside the interpretation. Changing which specimens enter the comparison can change the observed agreement. For example, a collection with many clearly positive and clearly negative specimens may show a different pattern from one concentrated near the category boundary. Describe the population to which you want the estimate to apply.

Describe the finding in plain language

Example wordingOf 50 paired specimens, the procedures agreed on 45. Relative to A, B agreed on 18 of 20 positive calls and 27 of 30 negative calls. Five pairs disagreed, in both directions.

The sample counts matter alongside the percentages. A high percentage based on a handful of specimens can be imprecise. A full comparison would consider how the specimens were selected and how uncertain the estimates are, particularly where a category is uncommon.

Member Learning

Plan a study and prepare the data

Planning a laboratory study and preparing your data takes you from a clear study question through data preparation to a summary you can explain.

Sources and related reading

Read about method comparison

Read about verification and validation

By Stephen MacDonald · Suggest a correction · Back to Find a guide