Many clinical studies ask a simple question: how well does a test separate patients with a disease from patients without it? Laboratory markers, imaging scores and risk scores all need this evaluation before they are used in practice. ROC curve analysis is the standard method to answer this question, because it summarizes the diagnostic accuracy of a test across all its possible cut-off values. A common mistake is to report only the area under the curve without a confidence interval, or to choose a cut-off value without a clear rule. This guide explains ROC curve analysis in simple steps, using one hypothetical clinical example from the beginning to the end.
Table of Contents
What Is ROC Curve Analysis?
A receiver operating characteristic (ROC) curve is a graph of the sensitivity (true positive rate) of a test on the y-axis against 1 − specificity (false positive rate) on the x-axis. Each point on the curve represents one possible cut-off value of the test (Nahm, 2022). A test with perfect discrimination reaches the upper left corner, while a useless test follows the diagonal line.
Sensitivity and Specificity
Sensitivity is the proportion of patients with the disease who have a positive test. Specificity is the proportion of patients without the disease who have a negative test. Lowering the cut-off increases sensitivity and decreases specificity, and the ROC curve shows this trade-off.
The Area Under the Curve (AUC)
The area under the curve summarizes the whole ROC curve in one number between 0.5 and 1.0. The AUC equals the probability that a randomly selected patient with the disease has a higher test value than a randomly selected patient without the disease (Hajian-Tilaki, 2013). As a general guide, an AUC of 0.7 to 0.8 is considered fair, 0.8 to 0.9 good and 0.9 or more excellent (Nahm, 2022).
Empirical and Smoothed ROC Curves
The empirical (non-parametric) ROC curve connects the observed points and makes no assumption about the distribution of the test values. In contrast, the smoothed (binormal) curve assumes normally distributed test values in both groups.
Importance of ROC Curve Analysis in Medical Research
Evaluation of New Diagnostic Tests
Before a new biomarker is introduced, researchers must show that it discriminates well between patients with and without the disease, independently of any single cut-off value.
Selection of the Optimal Cut-off Value
Clinicians need a practical threshold, such as “positive if the value is 0.5 ng/mL or more”. The ROC curve shows every candidate cut-off, so the choice becomes transparent and reproducible.
Comparison of Two or More Tests
When two tests are measured in the same patients, their AUCs can be compared statistically. This comparison shows whether a new test adds real diagnostic value over the test already in use.
Applications of ROC Curve Analysis in Clinical Studies
Diagnostic Accuracy Studies
Most diagnostic accuracy studies compare an index test with a reference standard, for example a biomarker against culture results or biopsy. These studies may follow a cross-sectional, cohort or case-control design, and ROC curve analysis is the main analysis in all of them.
Prediction Models and Risk Scores
The discrimination of a prediction model, such as one built with logistic regression, is usually reported as the AUC, also called the C statistic. Therefore, ROC curve analysis is used to compare clinical risk scores and to validate them in new populations.
Screening and Prognosis
Screening programs need tests with high sensitivity, while confirmatory tests need high specificity. In addition, ROC curves evaluate prognostic markers, such as markers of mortality after surgery.
Assumptions of ROC Curve Analysis
The following conditions should be met before performing ROC curve analysis:
- A valid reference standard: the true disease status of every patient is defined by an accepted reference test, which is performed independently of the index test.
- A binary outcome: patients are classified as having or not having the disease. Outcomes with more than two categories need special extensions of the method.
- An ordinal or continuous test: the index test produces a score or a measurement that can be ranked.
- Independent observations: each patient contributes one test value. Repeated measurements or several lesions per patient require clustered methods.
- A representative sample: the patients represent the clinical population in which the test will be used, which limits spectrum bias.
- An adequate sample size: enough patients with and without the disease are included to estimate the AUC precisely. Our guide to sample size explains this planning step.
How to Perform ROC Curve Analysis: A Clinical Example
Consider a hypothetical study of 200 adults who presented to the emergency department with fever. Serum procalcitonin (PCT, ng/mL) and C-reactive protein (CRP, mg/L) were measured at presentation. The reference standard was a confirmed bacterial infection, which was present in 80 patients (40%) and absent in 120 patients. The data were created for teaching purposes, and every value below was calculated from these data.
Step 1: Describe the Test Values in Each Group
Both skewed markers are presented as median and interquartile range (IQR). Both were significantly higher in patients with bacterial infection (Mann-Whitney U test, p < 0.001 for both).
| Marker | Bacterial infection (n = 80), median (IQR) | No bacterial infection (n = 120), median (IQR) |
|---|---|---|
| PCT (ng/mL) | 1.50 (0.71 to 3.58) | 0.23 (0.12 to 0.44) |
| CRP (mg/L) | 100.5 (66.8 to 163.0) | 50.0 (34.8 to 78.8) |
Step 2: Calculate Sensitivity and Specificity at Several Cut-off Values
A patient was classified as positive when the PCT value was equal to or higher than the cut-off. The table below shows how sensitivity falls and specificity rises as the cut-off increases:
| PCT cut-off (ng/mL) | Sensitivity (%) | Specificity (%) | Youden index | LR+ | LR− |
|---|---|---|---|---|---|
| ≥ 0.25 | 95.0 | 53.3 | 0.483 | 2.04 | 0.09 |
| ≥ 0.50 | 83.8 | 79.2 | 0.629 | 4.02 | 0.21 |
| ≥ 0.61 | 81.3 | 86.7 | 0.679 | 6.09 | 0.22 |
| ≥ 1.00 | 67.5 | 91.7 | 0.592 | 8.10 | 0.35 |
| ≥ 2.00 | 38.8 | 98.3 | 0.371 | 23.25 | 0.62 |
Step 3: Plot the ROC Curve
Joining the points of all cut-offs for PCT and CRP gives two ROC curves. The PCT curve lies closer to the upper left corner than the CRP curve, which already suggests better discrimination. A clear figure is an important part of data visualization in diagnostic studies.
Step 4: Calculate the AUC and Its 95% CI
The AUC of PCT was 0.899 (standard error 0.022, 95% CI, 0.855 to 0.942). The confidence interval was calculated with the non-parametric DeLong method (Nahm, 2022). The test of the null hypothesis AUC = 0.5 gave p < 0.001, so PCT discriminated significantly better than chance. For CRP, the AUC was 0.783 (95% CI, 0.719 to 0.846, p < 0.001).
Notably, the AUC is closely related to the Mann-Whitney U test. For PCT, U = 8,626.5, and U ÷ (80 × 120) = 8,626.5 ÷ 9,600 = 0.899, which is exactly the AUC.
Step 5: Select the Optimal Cut-off with the Youden Index
The Youden index equals sensitivity + specificity − 1. In this ROC curve analysis, its maximum value (J = 0.679) was reached at PCT ≥ 0.61 ng/mL, with a sensitivity of 81.3% and a specificity of 86.7% (Step 2 table). However, the Youden index gives equal weight to false positives and false negatives. When missing the disease is more harmful, a lower cut-off with higher sensitivity can be chosen.
Step 6: Build the 2 × 2 Table at the Selected Cut-off
At the cut-off of 0.61 ng/mL, the results were as follows:
| PCT result | Bacterial infection | No bacterial infection | Total |
|---|---|---|---|
| Positive (≥ 0.61) | 65 (TP) | 16 (FP) | 81 |
| Negative (< 0.61) | 15 (FN) | 104 (TN) | 119 |
| Total | 80 | 120 | 200 |
Sensitivity was 65 ÷ 80 = 81.3% (95% CI, 71.3 to 88.3), and specificity was 104 ÷ 120 = 86.7% (95% CI, 79.4 to 91.6). The confidence intervals were calculated with the Wilson method, which is based on the binomial distribution. Moreover, the positive predictive value (PPV) was 80.2% (95% CI, 70.3 to 87.5), the negative predictive value (NPV) was 87.4% (95% CI, 80.2 to 92.2) and the overall accuracy was 84.5%. The positive likelihood ratio was 6.09, and the negative likelihood ratio was 0.22. Importantly, PPV and NPV depend on the prevalence (40% here), so they change when the test is used in a different population.
Step 7: Compare Two ROC Curves with the DeLong Test
Because PCT and CRP were measured in the same patients, the two AUCs are correlated. Therefore, the paired DeLong test should be used. The difference in AUC was 0.116 (95% CI, 0.039 to 0.194, z = 2.93, p = 0.003). Thus, PCT showed significantly better diagnostic accuracy than CRP in this hypothetical sample.
How to Report ROC Curve Analysis Results in a Manuscript
The Methods section should describe the reference standard, the method for the AUC confidence interval, the rule for the optimal cut-off and the software. The Results section should present the significant findings, as shown below:
Diagnostic Accuracy of Procalcitonin and C-Reactive Protein for Bacterial Infection
Procalcitonin discriminated significantly between patients with and without bacterial infection (AUC = 0.899, 95% CI, 0.855 to 0.942, p < 0.001). At the optimal cut-off of 0.61 ng/mL, sensitivity was 81.3% (95% CI, 71.3 to 88.3) and specificity was 86.7% (95% CI, 79.4 to 91.6), with a PPV of 80.2% and an NPV of 87.4%. Moreover, C-reactive protein showed significant discrimination (AUC = 0.783, 95% CI, 0.719 to 0.846, p < 0.001). Notably, the AUC of procalcitonin was significantly higher than that of C-reactive protein (difference = 0.116, 95% CI, 0.039 to 0.194, p = 0.003). (Table 2)
Notably, the clinical meaning of these values and comparisons with previous studies belong to the Discussion section, not to the Results. The STARD guidelines also list the items that a diagnostic accuracy report should include.
Common Mistakes When Using ROC Curve Analysis
Reporting the AUC Without a Confidence Interval
An AUC of 0.80 from 40 patients is much less precise than the same AUC from 400 patients. Therefore, the 95% CI should always accompany the AUC, and a significant p-value against 0.5 is not enough on its own.
Choosing the Cut-off Without a Pre-specified Rule
Selecting the cut-off that gives the best result in the same data overestimates the performance of the test. The rule, such as the Youden index or a minimum sensitivity of 90%, should be stated in advance, and the cut-off should be validated in an independent sample.
Comparing Paired Curves with an Unpaired Test
Tests performed on the same patients are correlated, so the DeLong test for correlated curves should be used (Step 7).
Interpreting PPV and NPV as Fixed Properties
Predictive values change with disease prevalence, so a high PPV in a hospital sample may become a low PPV in primary care.
Ignoring the Direction of the Test
Some markers, such as albumin, are lower in diseased patients. If the software assumes that higher values indicate disease, the AUC falls below 0.5, so the correct direction should be selected before running the ROC curve analysis.
Running ROC Curve Analysis in SPSS and R
In SPSS, choose Analyze → Classify → ROC Curve. Move the test variable (PCT) to Test Variable and the reference standard to State Variable, and enter the value that indicates disease (for example 1). Then select With diagonal reference line, Standard error and confidence interval and Coordinate points of the ROC Curve. In R, the pROC package gives the AUC with the DeLong confidence interval in one line: ci.auc(roc(infection ~ pct, data = d)). Our beginner’s guide to R explains how to start.
Conclusion
ROC curve analysis is the standard method to evaluate the diagnostic accuracy of a continuous test. The AUC with its 95% CI summarizes discrimination, the Youden index or a pre-specified rule identifies a practical cut-off, and the DeLong test compares tests measured in the same patients. For binary outcomes, you can also read our guide to the odds ratio vs relative risk. If you need support in analyzing your diagnostic data or writing your Results section, the StatisMed team of biostatisticians provides statistical analysis services for medical studies. You can request a free quote today.
Frequently Asked Questions
1. What is a good AUC value?
An AUC of 0.5 indicates no discrimination, and 1.0 indicates perfect discrimination. In general, values of 0.7 to 0.8 are fair, 0.8 to 0.9 are good, and 0.9 or more are excellent. However, the required value depends on the clinical purpose of the test.
2. How many patients do I need for ROC curve analysis?
The sample size depends on the expected AUC, the desired width of the confidence interval and the ratio of patients with and without the disease. Both groups should be large enough to estimate sensitivity and specificity precisely.
3. Can I use ROC curve analysis for a logistic regression model?
Yes. The predicted probabilities from the model are used as the test variable. The resulting AUC, or C statistic, describes how well the model discriminates between patients with and without the outcome.
References
Hajian-Tilaki, K. (2013). Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation. Caspian Journal of Internal Medicine, 4(2), 627–635. Retrieved from https://pmc.ncbi.nlm.nih.gov/articles/PMC3755824/
Nahm, F. S. (2022). Receiver operating characteristic curve: Overview and practical use for clinicians. Korean Journal of Anesthesiology, 75(1), 25–36. https://doi.org/10.4097/kja.21209
