Reading a Method-Comparison Study Without a Statistics Background
The Diagnostic Literacy Series · Part 3
A practical guide to judging whether a method-comparison study supports the claims made about a diagnostic test — without a statistics background.
Independent educational information from The Aesculapius Foundation. This article does not endorse any manufacturer, product, laboratory, or testing platform.
Every new analyzer, assay, or point-of-care device arrives with a method-comparison study: a report claiming the new method agrees closely enough with an established reference method to be trusted. Reading that report critically does not require a statistics degree. It requires knowing which five questions to ask, in order, before accepting the manufacturer’s summary.
A method-comparison study answers one narrow question: does this method agree with a reference method, in these samples, under these conditions. It is not automatically true everywhere else.
Question 1: Was the comparison method actually a valid reference?
A method-comparison study is only as strong as the method it compares against. The reference method should itself be well-characterized, ideally an established laboratory method with known performance, not simply “the practice’s usual analyzer.” If a new method is compared only against another unvalidated device, agreement tells you the two devices behave similarly — not that either is accurate.
Ask: what was the comparator, and is its own performance documented? A study that compares a new point-of-care device against a reference laboratory method using standardized, accredited procedures carries more weight than one comparing it against another point-of-care device with no independent characterization.
Question 2: Were the samples representative of your patients?
Performance measured in a narrow, controlled sample set does not automatically transfer to a general clinical population. Look for:
- Species and breed range. A method validated only in adult mixed-breed dogs may behave differently in cats, in geriatric patients, or in a breed with a distinct hematologic or biochemical profile.
- Disease-state range. A study run entirely on healthy animals will not reveal how the method performs at abnormal, clinically relevant concentrations — which is usually exactly where the result matters most.
- Sample size. A handful of paired samples cannot support a strong claim of agreement. Look for how many samples were used and whether the range of values was wide enough to be clinically meaningful.
Question 3: What statistic did they use, and does it fit the question?
Three tools appear repeatedly in method-comparison literature, and each answers a different question. None of them requires you to run the calculation yourself — only to recognize what each one can and cannot tell you.
| Tool | What it shows | What it does not show |
|---|---|---|
| Correlation coefficient (r) | Whether two methods move together across a range of values | Whether they agree numerically — two methods can correlate strongly while one consistently reads higher than the other |
| Bland–Altman analysis | The average difference (bias) between methods and how much that difference varies (limits of agreement) | Whether the bias is clinically acceptable — that judgment requires a defined allowable-error threshold, which the study should state separately |
| Passing–Bablok or Deming regression | Systematic bias that changes across the measurement range (proportional bias), which ordinary linear regression can miss | Random, sample-to-sample scatter on its own — it should be read alongside Bland–Altman, not instead of it |
A study that reports only a correlation coefficient and calls the method “highly accurate” has not actually demonstrated agreement. A high correlation is easy to achieve and easy to misread; it is one of the most common ways a weak method-comparison study is dressed up to look strong.
Question 4: Is there a stated allowable-error threshold?
A difference between two methods only matters if it is large enough to change a clinical decision. A rigorous study defines, before collecting data, how much disagreement is acceptable for the analyte in question — often drawn from total allowable error guidelines or biological variation data — and then reports whether the observed bias and scatter fall inside that threshold.
A study that reports only “close agreement” or “acceptable correlation” without stating what threshold was used is asking you to accept the authors’ subjective judgment rather than a pre-specified, checkable standard. This is one of the fastest ways to separate a rigorous validation from a promotional one.
Question 5: Who ran the study, and does the paper say so plainly?
Manufacturer-funded and manufacturer-employed-author studies are not automatically wrong, but they carry a different evidentiary weight than independent replication, and a trustworthy paper says so directly rather than burying it in an acknowledgments footnote. When The Aesculapius Foundation discusses individual studies in this series, it labels the author relationship explicitly — independent academic, academic–industry hybrid, or manufacturer-employed — for exactly this reason: the label changes how much independent replication should be expected before the claim is treated as settled.
A five-question checklist to keep at hand:
- Was the reference method itself validated and well-characterized?
- Did the sample set include the species, disease states, and value range relevant to your patients?
- Did the study use Bland–Altman or regression analysis appropriate to detecting bias — not correlation alone?
- Was a specific allowable-error threshold stated and checked against, rather than a subjective “acceptable agreement”?
- Is the author relationship to the manufacturer disclosed, and has the finding been independently replicated?
What to do when the evidence is thin
Thin evidence does not automatically mean a method is unreliable — it means the claim has not yet been adequately tested for your use case. In that situation, a practical response is to run a small internal comparison before relying on the method for a consequential decision: collect paired samples across a realistic value range, compare them against the practice’s existing trusted method or a reference laboratory, and look specifically for consistent bias rather than isolated outliers.
This does not replace formal validation, and it will not detect every problem a properly powered study would find. But it can catch the most common and most consequential failure — a method that runs consistently high or low across the range your patients actually occupy — before that bias affects a treatment decision.
Where The Aesculapius Foundation fits
The Foundation’s Diagnostic Evaluation Program applies this same framework to real-world veterinary point-of-care testing: appropriate reference methods, representative sample sets, Bland–Altman and regression analysis where appropriate, pre-specified allowable-error thresholds, and transparent disclosure of every author’s relationship to the manufacturer whose product is being studied.
Help expand independent diagnostic evaluation
Charitable support helps the Foundation fund independent method-comparison studies and translate the results into plain-language guidance for practicing veterinarians. Support the Foundation’s work.
Frequently asked questions
Is a high correlation coefficient enough to trust a new diagnostic method?
No. A high correlation shows two methods move together across a range of values; it does not show whether they agree numerically. A method can correlate strongly with a reference method while consistently reading higher or lower — a systematic bias that correlation alone will not reveal.
What is the difference between Bland–Altman analysis and regression analysis?
Bland–Altman analysis shows the average difference between two methods and how much that difference varies across samples. Regression methods such as Passing–Bablok or Deming can reveal whether the bias changes at different concentrations — something a single average difference can miss. A thorough method-comparison study typically reports both.
Does manufacturer funding automatically make a study unreliable?
No, but it changes how much weight the finding should carry until independently replicated. A trustworthy paper discloses the funding source and author affiliations plainly, and independent replication strengthens confidence in a manufacturer-funded finding.
What should a practice do if it cannot find a rigorous method-comparison study for a device it is considering?
Ask the manufacturer directly for the underlying data and statistical methods used, look for independent literature, and consider running a small internal comparison against a trusted method before relying on the new device for consequential decisions.
Further reading
- American Society for Veterinary Clinical Pathology — Quality Assurance and Laboratory Standards Guidelines
- World Organisation for Animal Health — Principles and Methods of Validation of Diagnostic Assays
Educational notice. The Aesculapius Foundation provides independent educational information and maintains separation from commercial interests. Nothing in this article endorses a product, manufacturer, laboratory, or vendor.
]]>