Estimating the clinical effect of incorporating a new risk marker into therapeutic decisions via a risk reclassification strategy

Introduction

Risk reclassification tables are becoming an increasingly popular approach for evaluating the incremental prognostic information provided by a new risk marker beyond an established set of risk markers [1-9]. These cross-tabulations of estimated absolute risk display the allocation of study subjects across clinically relevant risk strata both before and after the addition of a new risk marker into the risk estimation process. Risk reclassification has been lauded as providing a better metric of the clinical utility of a new risk marker than more traditional measures such as p-values, effect sizes, and c-statistics, as reclassification tables more directly reflect the possible impact of a new marker on therapeutic decision making and possibly clinical outcomes [5,6,8,10-14]. In particular, assuming there exists a set of clinically relevant risk strata and sensible strata-specific treatment strategies, risk reclassification tables reveal how often a treatment strategy would change after incorporating the new risk marker into the risk estimation/treatment decision process. Furthermore, metrics such as the Net Reclassification Improvement (NRI) index have been developed for evaluating the appropriateness of risk reclassifications [1,7]. The NRI estimates the degree of "correct" reclassification after the addition of a new risk marker into risk estimation,with occurrence (or not) of the primary clinical outcome serving as the gauge for correctness [1,5,7].

Arguably, the ultimate clinical utility of a new risk marker is best judged by its ability to more appropriately guide therapeutic decision making, and consequently, to improve patient outcomes compared to a therapeutic strategy not guided by the new marker [3,13,15-17]. Though reclassification tables and the associated NRI are valuable for determining the incremental prognostic worth of a new risk marker, these approaches do not measure a new marker's potential impact on improving patient outcomes. Ideally, the value of incorporating a new risk marker into treatment decisions would be judged within the context of a randomized clinical trial where patients are randomized to management algorithms with or without guidance from the new risk marker [15,17]. Given the large and ever-increasing number of new markers requiring evaluation and the expected small clinical effects of incorporating a new risk marker into patient management strategies, randomized trials evaluating whether marker-guided patient management improves outcomes would necessarily be large, numerous, and costly; thus, few such trials are likely to be conducted [3,18].

Accordingly, the current report describes a quantitative method for indirectly estimating the effect of incorporating a new risk marker into therapeutic decisions via a risk reclassification strategy on clinical outcomes. In particular, the current investigation shows that given a set of clinically relevant risk strata, if reasonable estimates for risk strata-specific event rates and treatment effects can be provided, then a simple formula derived from a risk reclassification table enables estimating the relative and absolute expected number of events prevented/postponed when incorporating a new risk marker into therapeutic decision making via a risk reclassification strategy. The following discussion describes the method and provides an example demonstrating its application.

Methods

The current report describes a method using risk reclassification tables to estimate the effect of a therapeutic strategy guided by a new risk marker on clinical outcomes in a risk reclassification context. The proposed method will be most valuable when a clinical risk prediction model is already an established component of clinical decision making, clearly defined risk strata for the clinical outcome of interest have been demarcated and sensible strata-specific treatment strategies can be identified, and a new risk marker is being considered for inclusion in the established risk prediction model [3,4,19]. For example, the Framingham Risk Score (FRS) is an established tool for estimating the risk of an initial cardiac event, criteria have been defined for labeling individuals as low, intermediate, or high risk based on the estimated likelihood of an initial cardiac event within the next 10 years, and there has been recent interest in adding new risk markers such as high-sensitivity C-reactive protein and coronary artery calcium scores into the FRS algorithm [3,20-27]. (Table 1) illustrates the typical layout of a 3-level risk reclassification table with pre-marker risk class identified on the vertical axis and post-marker risk class identified on the horizontal axis.

Given a risk reclassification table generally adhering to the principles described above, practical application of the method described herein relies on five fundamental assumptions: (1) therapeutic intensity should be commensurate with clinical event risk; (2) post-marker risk strata-specific event rates can be estimated; (3) estimated risk strata-specific treatment effects are available; (4) relative risk reductions with a particular treatment intensity are independent of baseline absolute risk level; and (5) estimated post-marker risk is a better estimator of actual risk than estimated pre-marker risk. The first assumption expresses an attempt to find an optimal balance between efficacy, safety, and cost when making therapeutic decisions [3,4,13,15,28-32]. Generally, the principle conveys that higher risk patients warrant more intense therapy as expected therapeutic benefits will be larger among this group (more events will be prevented), thereby justifying the increased likelihood of side effects and higher costs typically accompanying more intense therapy. Conversely, lower risk patients receive less intense therapy, as possible side effects and costs should be minimized among this group with less expected therapeutic benefit. Though underlying risk can be expressed on a continuous scale as depicted in the (Figure 1), treatment decisions guided by estimated risk must necessarily be based on risk categories to facilitate practical application. The (Figure 1) designates three levels of risk and accordingly, three levels of treatment intensity. The second assumption for practical application of the method–that post-marker risk strata event rates can be estimated–is typically trivial, as these event rates can be calculated from the data used to construct the reclassification table at-hand. The third assumption–that risk strata-specific treatment effect estimates are available–relies on results from completed randomized clinical trials incorporating a similar clinical outcome and patient population as used to develop the risk reclassification table. Importantly, the number of estimated treatment effects must equal the number of risk strata. For example, with three risk classes, treatment effects must be available or reasonably estimated for three different treatment intensities (e.g., high dose, low dose, no treatment). The fourth assumption asserts that relative treatment effects for specific treatment intensities are constant across the entire spectrum of baseline risk; an associated assertion is that absolute treatment effects are positively associated with baseline risk [33]. Lastly, the fifth assumption claims that the new risk marker is a valid predictor of risk, thus improving risk estimation. Under these assumptions, the fundamental proposition of the proposed method is that (1) treatment decisions are based on estimated risk, (2) estimated risk is more accurate when including the new risk marker in the risk estimation process, and thus, (3) therapeutic decision making is enhanced when including the new risk marker in therapeutic decisions.

Under the aforementioned assumptions, the following components are needed in order to estimate the effect of incorporating a new risk marker into therapeutic decisions via a risk reclassification strategy: (1) a k x k risk reclassification table with k risk levels such as shown in (Table 1) for k=3, with observed cell counts for the k² cells; (2) estimated event rates within each of the k post-marker risk strata; and (3) estimated treatment effects for all k risk strata. Estimated event rates and treatment effects should preferably be on identical time scales (e.g., annual event rates and annual relative risk reductions). The following notation will be used to represent cell counts, post-marker risk strata event rates, and risk strata-specific treatment effects (with reference to Table 1). For cell counts, the first subscript denotes the pre-marker risk class and the second subscript denotes the post-marker risk class (Low, Intermediate, or High):

N_LL = Number of subjects in LOW-LOW cell (cell LL in Table 1),
N_LI = Number of subjects in LOW-INTERMEDIATE cell (LI),
N_LH = Number of subjects in LOW-HIGH cell (LH),
N_IL = Number of subjects in INTERMEDIATE-LOW cell (IL),
N_II = Number of subjects in INTERMEDIATE-INTERMEDIATE cell (II),
N_IH = Number of subjects in INTERMEDIATE-HIGH cell (IH),
N_HL = Number of subjects in HIGH-LOW cell (HL),
N_HI = Number of subjects in HIGH-INTERMEDIATE cell (HI),
N_HH = Number of subjects in HIGH-HIGH cell (HH);
R_L = Event rate in post-marker LOW RISK group,
R_I = Event rate in post-marker INTERMEDIATE RISK group,
R_H = Event rate in post-marker HIGH RISK group;
T_L = treatment effect in LOW RISK group (lowest treatment intensity),
T_I = treatment effect in INTERMEDIATE RISK group (intermediate treatment intensity),
T_H = treatment effect in HIGH RISK group (highest treatment intensity).

Then, the relative effect of incorporating a new risk marker into therapeutic decisions via a risk reclassification strategy can be expressed as A/B, where

A = (N_LL * R_L * T_L) + (N_LI * R_I * T_I) + (N_LH * R_H * T_H) + (N_IL * R_L * T_L) + (N_II * R_I * T_I) + (N_IH * R_H * T_H) + (N_HL * R_L * T_L) + (N_HI * R_I * T_I) + (N_HH * R_H * T_H), and
B = (N_LL * R_L * T_L) + (N_LI * R_I * T_L) + (N_LH * R_H * T_L) + (N_IL * R_L * T_I) + (N_II * R_I * T_I) + (N_IH * R_H * T_I) + (N_HL * R_L * T_H) + (N_HI * R_I * T_H) + (N_HH * R_H * T_H).

The absolute number of events prevented using the risk reclassification strategy is simply the difference between expressions (B–A).

Note that both expressions A and B consist of nine 3-item terms which provide estimates of the expected number of events over a specified time interval in each of the nine cells of the 3 by 3 risk reclassification table (Table 1). Expression A is the expected number of events when the therapeutic strategy is based on risk strata incorporating the new risk marker into the estimated event risk (post-marker), while expression B is the expected number of events when the therapeutic strategy is based on risk strata not incorporating the new risk marker into the estimated event risk (pre-marker). Of note, expression A could be rewritten as a function of the post-marker marginal totals, but the expanded expression is maintained here to facilitate comparing terms amongst expressions. Discrepancies between the two expressions lie strictly in the terms reflecting the off-diagonal cells (the reclassified subjects).

Results

The following example illustrates an application of the approach. The clinical scenario involves 3739 individuals with known or suspected coronary artery disease undergoing positron emission tomography (PET) imaging for the evaluation of myocardial perfusion subsequently followed for cardiac death [34]. Pre-marker risk estimation is based on a collection of variables including demographics, physical examination findings, cardiovascular risk factors, and medical history as previously described [34]. The size of myocardial perfusion defects as measured by PET serves as the new risk marker considered incrementally to pre-marker factors. The dependent outcome for this exercise is cardiac death at 2.5 years (binary), and low, intermediate, and high risk are defined as <1%, 1-3%, and >3% estimated cardiac death risk at 2.5 years. The risk reclassification table incorporating these risk strata is displayed in (Table 2). Of note, 14.1% of study subjects were reclassified after adding the new marker (PET perfusion defects) to the pre-marker risk model; the reclassification rate was higher in the pre-marker intermediate risk class (25.5%) than in the low (3.8%) or high (15.7%) risk classes. The c-statistic from a logistic regression model increased from 0.795 to 0.819 after adding the new marker to the risk model. The Net Reclassification Improvement was 11.5%.

Observed event rates at 2.5 years were 0.25%, 2.02%, and 7.52% across post-marker low, intermediate, and high risk groups, respectively. Furthermore, suppose that administered treatments across high, intermediate, and low risk strata prevent 50%, 25%, and 0% of 2.5-year events respectively. In the context of the current example, these treatment regimens may represent mechanical coronary artery revascularization procedures for high risk patients (50% relative risk reduction), aggressive pharmacological therapy for intermediate risk patients (25%), and no treatment for low risk patients (0%), respectively. Under these assumptions,
(Table 2) displays the estimated number of events per cell if patients are not treated, treated based on pre-marker risk strata, or treated based on post-marker risk strata, respectively. Using the equations defined above, the expected number of events in 2.5 years using pre-marker risk strata to guide treatment is:

B = 3.5 + 1.1 + 0.2 + 0.4 + 13.0 + 5.9 + 0.0 + 1.8 + 35.4 = 61.3.

Likewise, the expected number of events in 2.5 years using post-marker risk strata to guide treatment is:

A = 3.5 + 0.8 + 0.1 + 0.5 + 13.0 + 4.0 + 0.0 + 2.7 + 35.4 = 60.0.

Thus, the relative reduction in events by incorporating the new risk marker into therapeutic decisions via a risk reclassification strategy is: A/B = 60.0/61.3 = 0.979, or a 2.1% relative reduction in events by using the risk reclassification treatment strategy (95% confidence interval [CI] based on 100 bootstrap samples: (1.2%, 3.5%)). Alternatively, on the absolute scale an expected 1.3 events (95% CI: 0.7, 2.2) would be prevented at 2.5 years by using the risk reclassification treatment strategy.

If these calculations were restricted to individuals reclassified upward by the new marker (cells LI, LH, and IH), that is, if post-marker treatment was changed (intensified) only among those individuals reclassified to a higher post-marker risk class, there would be a 32.6% (95% CI: 31.0%, 34.3%) expected reduction in events among up-classified individuals (2.4 events prevented [95% CI: 1.2, 3.3]). However, in the current example only 160 of 3739 (4.3%) patients were up-classified. Furthermore, if these calculations were restricted to individuals at intermediate pre-marker risk, there would be a 10.3% (5.1%, 14.8%) expected relative reduction in events using a risk reclassification strategy among pre-marker intermediate risk patients (1.8 events prevented [95% CI: 0.9, 2.5]).

Discussion

Risk reclassification tables are a straightforward approach to understanding the effect of a new risk marker on changes in estimated risk and potentially therapeutic management. The Net Reclassification Improvement (NRI) index, an easyto-understand metric accompanying risk reclassification tables, is gaining popularity as a means of evaluating the appropriateness of risk reclassifications. Unfortunately, NRI values provide limited information for evaluating the possible effect of incorporating a new risk marker into therapeutic decisions on preventing clinical outcomes. Evaluating the clinical effect of a therapeutic paradigm guided by a new risk marker involves comparing two therapeutic algorithms: one based on estimated risk (and associated treatment) incorporating the new risk marker, and a second based on estimated risk (and associated treatment) not incorporating the new risk marker. Ideally, the value of incorporating a new risk marker into therapeutic decision making via a risk reclassification approach would be evaluated within the context of a randomized clinical trial. However, given the anticipated small clinical effects of a therapeutic management strategy incorporating risk reclassification and the enormous number of new risk markers requiring evaluation, such trials would be numerous, cumbersome to conduct, and thus few are likely to ever be performed. The approach described herein allows indirect estimation of the effect of a clinical strategy incorporating a new risk marker in a risk reclassification context. The general proposition of the method is that the new risk marker enables better estimation of subsequent event risk, allowing improved guidance of therapeutic management with a more favorable balance of efficacy, safety, and cost relative to a therapeutic strategy not guided by the new risk marker. The proposed method is simple to execute provided three pieces of information are available: (1) an appropriate risk reclassification table; (2) estimated event rates within post-marker risk strata; and (3) estimated treatment effects within risk strata for various treatment intensities.

The method described here complements and extends other approaches for evaluating the clinical value of a new risk marker such as p-values, effect sizes, c-statistics, and the NRI. Previous approaches have been criticized for being insensitive to important prognostic contributions from new risk markers and/or providing minimal clinical utility [1,14,29,34-36]. The current method is unique in attempting to estimate, albeit indirectly, the effect of a marker-guided treatment strategy guided by risk classes (and risk reclassification). The method relies on the oft-stated axiom that treatment intensity should parallel risk of untoward events. Under this axiom, risk classes determine treatment intensity; thus a change in risk class signifies a change in treatment intensity and as such, a change in event risk. The approach applies treatment effect estimates from randomized clinical trials and/or meta-analyses in order to estimate anticipated event reductions (increases) when incorporating a risk reclassification treatment strategy into clinical practice. The method is straightforward to apply conditional on an appropriate, empirically-derived risk reclassification table and a set of sensible risk strata-specific treatment effect estimates.

Several general observations can be noted from the exercise reported here. First, the extent of risk reduction (amplification) observed with a risk reclassification strategy depends partially on the magnitude of reclassification, as only reclassified individuals would receive a different therapeutic regimen then they otherwise would have received. Indeed, any observed benefit of a marker-guided strategy in this risk reclassification context is directly attributable to the more aggressive treatment of higher risk patients whose initial risk was underestimated using exclusively pre-marker information. With lower reclassification rates the relative effect will tend toward 1.00, as non-reclassified individuals will contribute equal numbers of expected events to both the numerator (A) and denominator (B), thus diluting the observed effect of the reclassification strategy [15]. Secondly, the balance between upward and downward reclassifications will directly affect the effect estimates since, based on the assumptions that treatment is efficacious and more intense treatment leads to greater risk reductions, upward reclassifications will always reduce the total number of expected events (as efficacious treatment would be initiated or intensified) and downward reclassifications will always increase the total number of expected events (as efficacious treatment would be halted or lessened). Indeed, under the assumption incorporated here that therapeutic intensity would be lessened among individuals whose estimated risk is lower after considering the new risk marker, a therapeutic management strategy based on risk reclassification may actually increase the expected number of events, particularly when the balance tends toward downward reclassifications. Some have questioned the appropriateness of lessening or withdrawing treatment among initially intermediate to high risk individuals whose estimated risk is lowered after considering a new risk marker [16,37]. Thus, recalculating effect estimates after excluding those cells in the reclassification table representing a downward shift in risk class may be a valuable supplemental exercise. Restricting calculations to upwardly-reclassified individuals will indubitably lead to stronger observed relative effects for the reclassification strategy, but at the expense of applying the strategy to only a fraction of the eligible study population (only 4.3% in the example reported here). Lastly, the estimated effects will clearly depend on the magnitude of variation across risk strata in both estimated event rates and treatment effects.

The current exercise also reveals the potential value of a risk reclassification strategy to guide therapeutic management irrespective of the new risk marker's effect on the marginal totals of the risk reclassification table. Many authors have stated a preference for new risk markers which place the majority of individuals into the extreme risk classes (and as few as possible in the intermediate risk class) where clinical decision making is supposedly less ambiguous [2,6,14,16,38]. Risk reclassification tables reported thus far have not approached this ideal. The current report reveals that greater post-marker allocation into more extreme risk categories is not necessarily a prerequisite for therapeutic strategies based on risk reclassification to be beneficial. Indeed, a therapeutic strategy based on risk reclassification provides value by improving risk estimation, thereby improving the efficiency of therapeutic management with a more appropriate balance of efficacy, safety, and cost–a standard not contingent upon maximizing the number of individuals at the risk extremes. Any individual whose estimated risk increases (and is estimated more accurately) by considering a new risk marker and thus is treated more aggressively should be considered more appropriately treated. Likewise, any individual whose estimated risk decreases and thus is treated less aggressively could be considered more appropriately treated from an aggregate perspective of efficacy, safety, and cost–notwithstanding the aforementioned concerns of reducing or withdrawing treatment from those with higher pre-marker risk.

Practical application of the method described here relies on certain assumptions whose validity will vary according to the clinical scenario. The dependence on clinically relevant cutpoints for defining risk strata and guiding treatment selection is an important limitation shared by other metrics evaluating the clinical merit of risk reclassification strategies [1,5,7,35]. Furthermore, sensible risk strata-specific treatment effects must be available and will typically be drawn from relevant randomized clinical trials and/or meta-analyses. Certainly, the estimated effect of a risk reclassification strategy for guiding therapeutic management will depend on the availability and validity of these estimated treatment effects. Sensitivity analyses may be helpful for evaluating the robustness of findings to different treatment effect estimates.As shown above, the method can easily be extended to certain subsets of interest by restricting calculations to the appropriate cells in the reclassification table. For instance, if only individuals at intermediate pre-marker risk or individuals with an upward shift in risk class were of interest, the calculations could easily be adjusted to focus on these patient subsets [5,13,39]. The proposed method will be more accurate when event rates within risk strata can be estimated independent of the treatment (s) being incorporated into the risk reclassification strategy. Thus, treatment-naïve event rate estimates may need to be back-calculated for optimal application of the method. Lastly, the proposed method only evaluates the effect of a risk reclassification strategy on clinical outcomes, without quantifying potential safety and cost implications of the strategy. A complete evaluation of a therapeutic strategy based on risk reclassification should consider clinical outcomes, safety, and costs in aggregate.

Conclusion

The current report describes a straightforward method for estimating the effect of incorporating a new risk marker into therapeutic decision making using a risk reclassification strategy. Researchers are encouraged to apply this method as an additional means of evaluating the clinical utility of a new risk marker.

Competing interests

The author declares that he has no competing interests.

Publication history

EIC: Jimmy Efird, East Carolina University, USA.
Received: 15-Oct-2013 Revised: 13-Nov-2013
Accepted: 15-Nov-2013 Published: 05-Dec-2013