Autor:in:
T. Ammer (Erlangen, DE)
Background
Precise reference intervals are essential for the interpretation of laboratory test results in medicine. Indirect methods leverage routine measurements containing a mixed distribution of non-pathological and pathological values to estimate reference intervals, rather than conducting a study with apparently healthy subjects (i.e. ‘direct’ method). In recent years, several such indirect methods have been developed. However, no standardized tool for the evaluation and comparison of indirect methods exists so far that can reveal the strengths and weaknesses of the different methods and guide algorithm selection and application.
Methods
We provide RIbench, a benchmarking suite that enables quantitative evaluation and comparison of existing and novel indirect methods. The benchmark contains simulated test sets for ten biomarkers mimicking real-world data (routine measurements). The non-pathological distribution of the biomarkers represent four common distribution types observed in laboratory practice: normal, skewed, heavily skewed, skewed-and-shifted. To identify limitations of the indirect methods, we added pathological distributions with varying location, extent of overlap, and fraction to the non-pathological distribution. Further, the sample size was varied to quantify the performance impact of the data set size. Overall, the benchmark suite contains 576 simulated tests sets per biomarker, 5,760 test sets in total. To evaluate the performance, we compute benchmark scores derived from the absolute z-score deviations between the estimated and true reference limits. We showcase the application of RIbench by evaluating five indirect methods, the Hoffmann method, and four modern approaches: TML, kosmic, TMC, and refineR. The results are compared against each another and a nonparametric direct method (N=120).
Results
For all methods, the pathological fraction had a strong influence on the results. Further, for TML, kosmic, TMC, and refineR, the sample size also strongly affected the performance. With a minimum sample size of 5,000 and a pathological fraction of up to 20%, these indirect methods still achieved results comparable or superior to the direct method.
Conclusions
We present RIbench, an open-source R-package that enables a quantitative and systematic evaluation and comparison of existing and novel indirect methods. Covering a variety of tests sets with varying difficulty, RIbench can serve as a valuable tool to reveal strengths and weaknesses, and enhance indirect methods, ultimately improving the estimation of reference intervals.