Brent A. Williams

Correspondence: Brent A. Williams bawilliams2@geisinger.edu

**Authors Affiliation**

Geisinger Health System, 100 North Academy Avenue, Danville, PA, USA.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

**Background:** Risk reclassification tables are an effective means of evaluating the added prognostic information provided by a new risk marker beyond an established set of risk markers. These cross-tabulations display the allocation of individuals into clinicallyrelevant risk strata both before and after the addition of a new risk marker into the risk estimation process. Risk reclassification tables have been lauded as providing a better metric of the clinical utility of a new risk marker than traditional metrics such as p-values, effect sizes, and c-statistics, as the extent of reclassification reveals how often therapeutic anagement may change when incorporating the new risk marker into risk estimation, and thus, therapeutic decision making. The ultimate clinical utility of new risk markers would be best judged within the context of randomized clinical trials. Unfortunately, given the anticipated small clinical effects of a marker-guided treatment strategy, and a large and ever-increasing number of markers requiring evaluation, such trials would necessarily be large, numerous, costly, and unlikely to ever be performed. Thus, novel methods for evaluating the potential impact of marker-guided therapeutic strategies on clinical outcomes should be sought.

**Methods:** Accordingly, the current report describes a quantitative method utilizing risk reclassification tables which enables estimating the effect of incorporating a new risk marker into therapeutic decisions on clinical outcomes. The proposed method employs (1) an appropriate risk reclassification table, (2) estimated event rates within risk strata, and (3) estimated treatment effects within risk strata to estimate the relative and absolute decrease in the expected number of events when incorporating a new risk marker into therapeutic decisions via a risk reclassification strategy. A real-life example demonstrates the approach.

**Conclusions:** The current report describes a straightforward method for indirectly estimating the potential clinical effect of incorporating a new risk marker into therapeutic decision making via a risk reclassification strategy. In the absence of randomized trials directly estimating the effect of a marker-guided strategy on clinical outcomes, the current method enables an indirect approach for anticipating the outcome of such trials.

**Keywords:** Risk, reclassification table, clinical prediction model, biomarker

Risk reclassification tables are becoming an increasingly popular approach for evaluating the incremental prognostic information provided by a new risk marker beyond an established set of risk markers [1-9]. These cross-tabulations of estimated absolute risk display the allocation of study subjects across clinically relevant risk strata both before and after the addition of a new risk marker into the risk estimation process. Risk reclassification has been lauded as providing a better metric of the clinical utility of a new risk marker than more traditional measures such as p-values, effect sizes, and c-statistics, as reclassification tables more directly reflect the possible impact of a new marker on therapeutic decision making and possibly clinical outcomes [5,6,8,10-14]. In particular, assuming there exists a set of clinically relevant risk strata and sensible strata-specific treatment strategies, risk reclassification tables reveal how often a treatment strategy would change after incorporating the new risk marker into the risk estimation/treatment decision process. Furthermore, metrics such as the Net Reclassification Improvement (NRI) index have been developed for evaluating the appropriateness of risk reclassifications [1,7]. The NRI estimates the degree of "correct" reclassification after the addition of a new risk marker into risk estimation,with occurrence (or not) of the primary clinical outcome serving as the gauge for correctness [1,5,7].

Arguably, the ultimate clinical utility of a new risk marker is best judged by its ability to *more appropriately guide therapeutic decision making*, and consequently, to *improve patient outcomes* compared to a therapeutic strategy not guided by the new marker [3,13,15-17]. Though reclassification tables and the associated NRI are valuable for determining the incremental prognostic worth of a new risk marker, these approaches do not measure a new marker's potential impact on improving patient outcomes. Ideally, the value of incorporating a new risk marker into treatment decisions would be judged within the context of a randomized clinical trial where patients are randomized to management algorithms with or without guidance from the new risk marker [15,17]. Given the large and ever-increasing number of new markers requiring evaluation and the expected small clinical effects of incorporating a new risk marker into patient management strategies, randomized trials evaluating whether marker-guided patient management improves outcomes would necessarily be large, numerous, and costly; thus, few such trials are likely to be conducted [3,18].

Accordingly, the current report describes a quantitative method for indirectly estimating the effect of incorporating a new risk marker into therapeutic decisions via a risk reclassification strategy on clinical outcomes. In particular, the current investigation shows that given a set of clinically relevant risk strata, if reasonable estimates for risk strata-specific event rates and treatment effects can be provided, then a simple formula derived from a risk reclassification table enables estimating the relative and absolute expected number of events prevented/postponed when incorporating a new risk marker into therapeutic decision making via a risk reclassification strategy. The following discussion describes the method and provides an example demonstrating its application.

The current report describes a method using risk reclassification tables to estimate the effect of a therapeutic strategy guided by a new risk marker on clinical outcomes in a risk reclassification context. The proposed method will be most valuable when a clinical risk prediction model is already an established component of clinical decision making, clearly defined risk strata for the clinical outcome of interest have been demarcated and sensible strata-specific treatment strategies can be identified, and a new risk marker is being considered for inclusion in the established risk prediction model [3,4,19]. For example, the Framingham Risk Score (FRS) is an established tool for estimating the risk of an initial cardiac event, criteria have been defined for labeling individuals as low, intermediate, or high risk based on the estimated likelihood of an initial cardiac event within the next 10 years, and there has been recent interest in adding new risk markers such as high-sensitivity C-reactive protein and coronary artery calcium scores into the FRS algorithm [3,20-27]. (Table 1) illustrates the typical layout of a 3-level risk reclassification table with pre-marker risk class identified on the vertical axis and post-marker risk class identified on the horizontal axis.

Table 1 : **Layout of a typical risk reclassification table using three risk categories.**

Given a risk reclassification table generally adhering to the principles described above, practical application of the method described herein relies on five fundamental assumptions: (1) therapeutic intensity should be commensurate with clinical event risk; (2) post-marker risk strata-specific event rates can be estimated; (3) estimated risk strata-specific treatment effects are available; (4) relative risk reductions with a particular treatment intensity are independent of baseline absolute risk level; and (5) estimated post-marker risk is a better estimator of actual risk than estimated pre-marker risk. The first assumption expresses an attempt to find an optimal balance between efficacy, safety, and cost when making therapeutic decisions [3,4,13,15,28-32]. Generally, the principle conveys that higher risk patients warrant more intense therapy as expected therapeutic benefits will be larger among this group (more events will be prevented), thereby justifying the increased likelihood of side effects and higher costs typically accompanying more intense therapy. Conversely, lower risk patients receive less intense therapy, as possible side effects and costs should be minimized among this group with less expected therapeutic benefit. Though underlying risk can be expressed on a continuous scale as depicted in the (Figure 1), treatment decisions guided by estimated risk must necessarily be based on risk categories to facilitate practical application. The (Figure 1) designates three levels of risk and accordingly, three levels of treatment intensity. The second assumption for practical application of the method–that post-marker risk strata event rates can be estimated–is typically trivial, as these event rates can be calculated from the data used to construct the reclassification table at-hand. The third assumption–that risk strata-specific treatment effect estimates are available–relies on results from completed randomized clinical trials incorporating a similar clinical outcome and patient population as used to develop the risk reclassification table. Importantly, the number of estimated treatment effects must equal the number of risk strata. For example, with three risk classes, treatment effects must be available or reasonably estimated for three different treatment intensities (e.g., high dose, low dose, no treatment). The fourth assumption asserts that relative treatment effects for specific treatment intensities are constant across the entire spectrum of baseline risk; an associated assertion is that absolute treatment effects are positively associated with baseline risk [33]. Lastly, the fifth assumption claims that the new risk marker is a valid predictor of risk, thus improving risk estimation. Under these assumptions, the fundamental proposition of the proposed method is that (1) treatment decisions are based on estimated risk, (2) estimated risk is more accurate when including the new risk marker in the risk estimation process, and thus, (3) therapeutic decision making is enhanced when including the new risk marker in therapeutic decisions.

Figure 1 :** The principle of basing treatment intensity on estimated event risk.**

Under the aforementioned assumptions, the following components are needed in order to estimate the effect of incorporating a new risk marker into therapeutic decisions via a risk reclassification strategy: (1) a *k* x *k* risk reclassification table with *k* risk levels such as shown in (Table 1) for *k*=3, with observed cell counts for the *k ^{2}* cells; (2) estimated event rates within each of the

N_{LL} = Number of subjects in LOW-LOW cell (cell LL in Table 1),

N_{LI} = Number of subjects in LOW-INTERMEDIATE cell (LI),

N_{LH} = Number of subjects in LOW-HIGH cell (LH),

N_{IL} = Number of subjects in INTERMEDIATE-LOW cell (IL),

N_{II} = Number of subjects in INTERMEDIATE-INTERMEDIATE cell (II),

N_{IH} = Number of subjects in INTERMEDIATE-HIGH cell (IH),

N_{HL} = Number of subjects in HIGH-LOW cell (HL),

N_{HI} = Number of subjects in HIGH-INTERMEDIATE cell (HI),

N_{HH} = Number of subjects in HIGH-HIGH cell (HH);

R_{L} = Event rate in post-marker LOW RISK group,

R_{I} = Event rate in post-marker INTERMEDIATE RISK group,

R_{H} = Event rate in post-marker HIGH RISK group;

T_{L} = treatment effect in LOW RISK group (lowest treatment intensity),

T_{I }= treatment effect in INTERMEDIATE RISK group (intermediate treatment intensity),

T_{H} = treatment effect in HIGH RISK group (highest treatment intensity).

Then, the relative effect of incorporating a new risk marker into therapeutic decisions via a risk reclassification strategy can be expressed as A/B, where

A = (N_{LL} * R_{L} * T_{L}) + (N_{LI} * R_{I }* T_{I}) + (N_{LH} * R_{H} * T_{H}) + (N_{IL} * R_{L} * T_{L}) + (N_{II} * R_{I} * T_{I}) + (N_{IH} * R_{H} * T_{H}) + (N_{HL} * R_{L} * T_{L}) + (N_{HI} * R_{I} * T_{I}) + (N_{HH} * R_{H} * T_{H}), and

B = (N_{LL} * R_{L} * T_{L}) + (N_{LI} * R_{I} * T_{L}) + (N_{LH} * R_{H} * T_{L}) + (N_{IL} * R_{L} * T_{I}) + (N_{II} * R_{I }* T_{I}) + (N_{IH} * R_{H} * T_{I}) + (N_{HL} * R_{L} * T_{H}) + (N_{HI} * R_{I} * T_{H}) + (N_{HH} * R_{H} * T_{H}).

The absolute number of events prevented using the risk reclassification strategy is simply the difference between expressions (B–A).

Note that both expressions A and B consist of nine 3-item terms which provide estimates of the expected number of events over a specified time interval in each of the nine cells of the 3 by 3 risk reclassification table (Table 1). Expression A is the expected number of events when the therapeutic strategy is based on risk strata incorporating the new risk marker into the estimated event risk (post-marker), while expression B is the expected number of events when the therapeutic strategy is based on risk strata not incorporating the new risk marker into the estimated event risk (pre-marker). Of note, expression A could be rewritten as a function of the post-marker marginal totals, but the expanded expression is maintained here to facilitate comparing terms amongst expressions. Discrepancies between the two expressions lie strictly in the terms reflecting the off-diagonal cells (the reclassified subjects).

The following example illustrates an application of the approach. The clinical scenario involves 3739 individuals with known or suspected coronary artery disease undergoing positron emission tomography (PET) imaging for the evaluation of myocardial perfusion subsequently followed for cardiac death [34]. Pre-marker risk estimation is based on a collection of variables including demographics, physical examination findings, cardiovascular risk factors, and medical history as previously described [34]. The size of myocardial perfusion defects as measured by PET serves as the new risk marker considered incrementally to pre-marker factors. The dependent outcome for this exercise is cardiac death at 2.5 years (binary), and low, intermediate, and high risk are defined as <1%, 1-3%, and >3% estimated cardiac death risk at 2.5 years. The risk reclassification table incorporating these risk strata is displayed in (Table 2). Of note, 14.1% of study subjects were reclassified after adding the new marker (PET perfusion defects) to the pre-marker risk model; the reclassification rate was higher in the pre-marker intermediate risk class (25.5%) than in the low (3.8%) or high (15.7%) risk classes. The c-statistic from a logistic regression model increased from 0.795 to 0.819 after adding the new marker to the risk model. The Net Reclassification Improvement was 11.5%.

Table 2 : **Risk reclassification table and expected number of events with and without incorporating a new risk marker into therapeutic management via a risk reclassification strategy.**

Observed event rates at 2.5 years were 0.25%, 2.02%, and 7.52% across post-marker low, intermediate, and high risk groups, respectively. Furthermore, suppose that administered treatments across high, intermediate, and low risk strata prevent 50%, 25%, and 0% of 2.5-year events respectively. In the context of the current example, these treatment regimens may represent mechanical coronary artery revascularization procedures for high risk patients (50% relative risk reduction), aggressive pharmacological therapy for intermediate risk patients (25%), and no treatment for low risk patients (0%), respectively. Under these assumptions,

(Table 2) displays the estimated number of events per cell if patients are not treated, treated based on pre-marker risk strata, or treated based on post-marker risk strata, respectively. Using the equations defined above, the expected number of events in 2.5 years using pre-marker risk strata to guide treatment is:

B = 3.5 + 1.1 + 0.2 + 0.4 + 13.0 + 5.9 + 0.0 + 1.8 + 35.4 = 61.3.

Likewise, the expected number of events in 2.5 years using post-marker risk strata to guide treatment is:

A = 3.5 + 0.8 + 0.1 + 0.5 + 13.0 + 4.0 + 0.0 + 2.7 + 35.4 = 60.0.

Thus, the relative reduction in events by incorporating the new risk marker into therapeutic decisions via a risk reclassification strategy is: A/B = 60.0/61.3 = 0.979, or a 2.1% relative reduction in events by using the risk reclassification treatment strategy (95% confidence interval [CI] based on 100 bootstrap samples: (1.2%, 3.5%)). Alternatively, on the absolute scale an expected 1.3 events (95% CI: 0.7, 2.2) would be prevented at 2.5 years by using the risk reclassification treatment strategy.

If these calculations were restricted to individuals reclassified upward by the new marker (cells LI, LH, and IH), that is, if post-marker treatment was changed (intensified) only among those individuals reclassified to a higher post-marker risk class, there would be a 32.6% (95% CI: 31.0%, 34.3%) expected reduction in events among up-classified individuals (2.4 events prevented [95% CI: 1.2, 3.3]). However, in the current example only 160 of 3739 (4.3%) patients were up-classified. Furthermore, if these calculations were restricted to individuals at intermediate pre-marker risk, there would be a 10.3% (5.1%, 14.8%) expected relative reduction in events using a risk reclassification strategy among pre-marker intermediate risk patients (1.8 events prevented [95% CI: 0.9, 2.5]).

Risk reclassification tables are a straightforward approach to understanding the effect of a new risk marker on changes in estimated risk and potentially therapeutic management. The Net Reclassification Improvement (NRI) index, an easyto-understand metric accompanying risk reclassification tables, is gaining popularity as a means of evaluating the appropriateness of risk reclassifications. Unfortunately, NRI values provide limited information for evaluating the possible effect of incorporating a new risk marker into therapeutic decisions on preventing clinical outcomes. Evaluating the clinical effect of a therapeutic paradigm guided by a new risk marker involves comparing two therapeutic algorithms: one based on estimated risk (and associated treatment) incorporating the new risk marker, and a second based on estimated risk (and associated treatment) not incorporating the new risk marker. Ideally, the value of incorporating a new risk marker into therapeutic decision making via a risk reclassification approach would be evaluated within the context of a randomized clinical trial. However, given the anticipated small clinical effects of a therapeutic management strategy incorporating risk reclassification and the enormous number of new risk markers requiring evaluation, such trials would be numerous, cumbersome to conduct, and thus few are likely to ever be performed. The approach described herein allows indirect estimation of the effect of a clinical strategy incorporating a new risk marker in a risk reclassification context. The general proposition of the method is that the new risk marker enables better estimation of subsequent event risk, allowing improved guidance of therapeutic management with a more favorable balance of efficacy, safety, and cost relative to a therapeutic strategy not guided by the new risk marker. The proposed method is simple to execute provided three pieces of information are available: (1) an appropriate risk reclassification table; (2) estimated event rates within post-marker risk strata; and (3) estimated treatment effects within risk strata for various treatment intensities.

The method described here complements and extends other approaches for evaluating the clinical value of a new risk marker such as p-values, effect sizes, c-statistics, and the NRI. Previous approaches have been criticized for being insensitive to important prognostic contributions from new risk markers and/or providing minimal clinical utility [1,14,29,34-36]. The current method is unique in attempting to estimate, albeit indirectly, the effect of a marker-guided treatment strategy guided by risk classes (and risk reclassification). The method relies on the oft-stated axiom that treatment intensity should parallel risk of untoward events. Under this axiom, risk classes determine treatment intensity; thus a change in risk class signifies a change in treatment intensity and as such, a change in event risk. The approach applies treatment effect estimates from randomized clinical trials and/or meta-analyses in order to estimate anticipated event reductions (increases) when incorporating a risk reclassification treatment strategy into clinical practice. The method is straightforward to apply conditional on an appropriate, empirically-derived risk reclassification table and a set of sensible risk strata-specific treatment effect estimates.

Several general observations can be noted from the exercise reported here. First, the extent of risk reduction (amplification) observed with a risk reclassification strategy depends partially on the magnitude of reclassification, as only reclassified individuals would receive a different therapeutic regimen then they otherwise would have received. Indeed, any observed benefit of a marker-guided strategy in this risk reclassification context is directly attributable to the more aggressive treatment of higher risk patients whose initial risk was underestimated using exclusively pre-marker information. With lower reclassification rates the relative effect will tend toward 1.00, as non-reclassified individuals will contribute equal numbers of expected events to both the numerator (A) and denominator (B), thus diluting the observed effect of the reclassification strategy [15]. Secondly, the balance between upward and downward reclassifications will directly affect the effect estimates since, based on the assumptions that treatment is efficacious and more intense treatment leads to greater risk reductions, upward reclassifications will always reduce the total number of expected events (as efficacious treatment would be initiated or intensified) and downward reclassifications will always increase the total number of expected events (as efficacious treatment would be halted or lessened). Indeed, under the assumption incorporated here that therapeutic intensity would be lessened among individuals whose estimated risk is lower after considering the new risk marker, a therapeutic management strategy based on risk reclassification may actually increase the expected number of events, particularly when the balance tends toward downward reclassifications. Some have questioned the appropriateness of lessening or withdrawing treatment among initially intermediate to high risk individuals whose estimated risk is lowered after considering a new risk marker [16,37]. Thus, recalculating effect estimates after excluding those cells in the reclassification table representing a downward shift in risk class may be a valuable supplemental exercise. Restricting calculations to upwardly-reclassified individuals will indubitably lead to stronger observed relative effects for the reclassification strategy, but at the expense of applying the strategy to only a fraction of the eligible study population (only 4.3% in the example reported here). Lastly, the estimated effects will clearly depend on the magnitude of variation across risk strata in both estimated event rates and treatment effects.

The current exercise also reveals the potential value of a risk reclassification strategy to guide therapeutic management irrespective of the new risk marker's effect on the marginal totals of the risk reclassification table. Many authors have stated a preference for new risk markers which place the majority of individuals into the extreme risk classes (and as few as possible in the intermediate risk class) where clinical decision making is supposedly less ambiguous [2,6,14,16,38]. Risk reclassification tables reported thus far have not approached this ideal. The current report reveals that greater post-marker allocation into more extreme risk categories is not necessarily a prerequisite for therapeutic strategies based on risk reclassification to be beneficial. Indeed, a therapeutic strategy based on risk reclassification provides value by improving risk estimation, thereby improving the efficiency of therapeutic management with a more appropriate balance of efficacy, safety, and cost–a standard not contingent upon maximizing the number of individuals at the risk extremes. Any individual whose estimated risk increases (and is estimated more accurately) by considering a new risk marker and thus is treated more aggressively should be considered more appropriately treated. Likewise, any individual whose estimated risk decreases and thus is treated less aggressively could be considered more appropriately treated from an aggregate perspective of efficacy, safety, and cost–notwithstanding the aforementioned concerns of reducing or withdrawing treatment from those with higher pre-marker risk.

Practical application of the method described here relies on certain assumptions whose validity will vary according to the clinical scenario. The dependence on clinically relevant cutpoints for defining risk strata and guiding treatment selection is an important limitation shared by other metrics evaluating the clinical merit of risk reclassification strategies [1,5,7,35]. Furthermore, sensible risk strata-specific treatment effects must be available and will typically be drawn from relevant randomized clinical trials and/or meta-analyses. Certainly, the estimated effect of a risk reclassification strategy for guiding therapeutic management will depend on the availability and validity of these estimated treatment effects. Sensitivity analyses may be helpful for evaluating the robustness of findings to different treatment effect estimates.As shown above, the method can easily be extended to certain subsets of interest by restricting calculations to the appropriate cells in the reclassification table. For instance, if only individuals at intermediate pre-marker risk or individuals with an upward shift in risk class were of interest, the calculations could easily be adjusted to focus on these patient subsets [5,13,39]. The proposed method will be more accurate when event rates within risk strata can be estimated independent of the treatment (s) being incorporated into the risk reclassification strategy. Thus, treatment-naïve event rate estimates may need to be back-calculated for optimal application of the method. Lastly, the proposed method only evaluates the effect of a risk reclassification strategy on clinical outcomes, without quantifying potential safety and cost implications of the strategy. A complete evaluation of a therapeutic strategy based on risk reclassification should consider clinical outcomes, safety, and costs in aggregate.

The current report describes a straightforward method for estimating the effect of incorporating a new risk marker into therapeutic decision making using a risk reclassification strategy. Researchers are encouraged to apply this method as an additional means of evaluating the clinical utility of a new risk marker.

The author declares that he has no competing interests.

EIC: Jimmy Efird, East Carolina University, USA.

Received: 15-Oct-2013 Revised: 13-Nov-2013

Accepted: 15-Nov-2013 Published: 05-Dec-2013

- Pencina MJ, D'Agostino RB, Sr. and Steyerberg EW.
**Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers**.*Stat Med.*2011;**30**:11-21. | Article | PubMed Abstract | PubMed Full Text - Pepe MS.
**Problems with risk reclassification methods for evaluating prediction models**.*Am J Epidemiol.*2011;**173**:1327-35. | Article | PubMed Abstract | PubMed Full Text - Wang TJ.
**Assessing the role of circulating, genetic, and imaging biomarkers in cardiovascular risk prediction**.*Circulation.*2011;**123**:551-65. | Article | PubMed Abstract | PubMed Full Text - Lloyd-Jones DM.
**Cardiovascular risk prediction: basic concepts, current status, and future directions**.*Circulation.*2010;**121**:1768-77. | Article | PubMed - Cook NR and Ridker PM.
**Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures**.*Ann Intern Med.*2009;**150**:795-802. | Article | PubMed Abstract | PubMed Full Text - Janes H, Pepe MS and Gu W.
**Assessing the value of risk predictions by using risk stratification tables**.*Ann Intern Med.*2008;**149**:751-60. | Article | PubMed Abstract | PubMed Full Text - Pencina MJ, D'Agostino RB, Sr., D'Agostino RB, Jr. and Vasan RS.
**Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond**.*Stat Med.*2008;**27**:157-72. | Article | PubMed - Cook NR.
**Use and misuse of the receiver operating characteristic curve in risk prediction**.*Circulation.*2007;**115**:928-35. | Article | PubMed - Ridker PM, Buring JE, Rifai N and Cook NR.
**Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: the Reynolds Risk Score**.*JAMA.*2007;**297**:611-9. | Article | PubMed - Romanens M, Ackermann F, Spence JD, Darioli R, Rodondi N, Corti R, Noll G, Schwenkglenks M and Pencina M.
**Improvement of cardiovascular risk prediction: time to review current knowledge, debates, and fundamentals on how to assess test characteristics**.*Eur J Cardiovasc Prev Rehabil.*2010;**17**:18-23. | Article | PubMed - Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ and Kattan MW.
**Assessing the performance of prediction models: a framework for traditional and novel measures**.*Epidemiology.*2010;**21**:128-38. | Article | PubMed Abstract | PubMed Full Text - Cooney MT, Dudina AL and Graham IM.
**Value and limitations of existing scores for the assessment of cardiovascular risk: a review for clinicians**.*J Am Coll Cardiol.*2009;**54**:1209-27. | Article | PubMed - Hlatky MA, Greenland P, Arnett DK, Ballantyne CM, Criqui MH, Elkind MS, Go AS, Harrell FE, Jr., Hong Y and Howard BV et al.
**Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association**.*Circulation.*2009;**119**:2408-16. | Article | PubMed Abstract | PubMed Full Text - Cook NR.
**Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve**.*Clin Chem.*2008;**54**:17-23. | Article | PubMed - Pletcher MJ and Pignone M.
**Evaluating the clinical utility of a biomarker: a review of methods for estimating health impact**.*Circulation.*2011;**123**:1116-24. | Article | PubMed Abstract | PubMed Full Text - O'Malley PG and Redberg RF.
**Risk refinement, reclassification, and treatment thresholds in primary prevention of cardiovascular disease: incremental progress but significant gaps remain**.*Arch Intern Med.*2010;**170**:1602-3. | Article | PubMed - Ioannidis JP and Tzoulaki I.
**What makes a good predictor?: the evidence applied to coronary artery calcium score**.*JAMA.*2010;**303**:1646-7. | Article | PubMed - Wang TJ.
**Usefulness of novel screening tests for cardiovascular disease**.*Arch Intern Med.*2011;**171**:284-5. | Article | PubMed - Gu W and Pepe MS.
**Estimating the capacity for improvement in risk prediction with a marker**.*Biostatistics.*2009;**10**:172-86. | Article | PubMed Abstract | PubMed Full Text - Ferket BS, Genders TS, Colkesen EB, Visser JJ, Spronk S, Steyerberg EW and Hunink MG.
**Systematic review of guidelines on imaging of asymptomatic coronary artery disease**.*J Am Coll Cardiol.*2011;**57**:1591-600. | Article | PubMed - Polonsky TS, McClelland RL, Jorgensen NW, Bild DE, Burke GL, Guerci AD and Greenland P.
**Coronary artery calcium score and risk classification for coronary heart disease prediction**.*JAMA.*2010;**303**:1610-6. | Article | PubMed Abstract | PubMed Full Text - Buckley DI, Fu R, Freeman M, Rogers K and Helfand M.
**C-reactive protein as a risk factor for coronary heart disease: a systematic review and meta-analyses for the U.S. Preventive Services Task Force**.*Ann Intern Med.*2009;**151**:483-95. | Article | PubMed - Ridker PM, Paynter NP, Rifai N, Gaziano JM and Cook NR.
**C-reactive protein and parental history improve global cardiovascular risk prediction: the Reynolds Risk Score for men**.*Circulation.*2008;**118**:2243-51. | Article | PubMed Abstract | PubMed Full Text - Wilson PW, Pencina M, Jacques P, Selhub J, D'Agostino R, Sr. and O'Donnell CJ.
**C-reactive protein and reclassification of cardiovascular risk in the Framingham Heart Study**.*Circ Cardiovasc Qual Outcomes.*2008;**1**:92-7. | Article | PubMed Abstract | PubMed Full Text - Lakoski SG, Cushman M, Blumenthal RS, Kronmal R, Arnett D, D'Agostino RB, Jr., Detrano RC and Herrington DM.
**Implications of C-reactive protein or coronary artery calcium score as an adjunct to global risk assessment for primary prevention of CHD**.*Atherosclerosis.*2007;**193**:401-7. | Article | PubMed - Cook NR, Buring JE and Ridker PM.
**The effect of including C-reactive protein in cardiovascular risk prediction models for women**.*Ann Intern Med.*2006;**145**:21-9. | Article | PubMed - Ridker PM.
**High-sensitivity C-reactive protein, inflammation, and cardiovascular risk: from concept to clinical practice to clinical benefit**.*Am Heart J.*2004;**148**:S19-26. | Article | PubMed - Cooney MT, Dudina A, D'Agostino R and Graham IM.
**Cardiovascular risk-estimation systems in primary prevention: do they differ? Do they make a difference? Can we see the future?***Circulation.*2010;**122**:300-10. | Article | PubMed - McGeechan K, Macaskill P, Irwig L, Liew G and Wong TY.
**Assessing new biomarkers and predictive models for use in clinical practice: a clinician's guide**.*Arch Intern Med.*2008;**168**:2304-10. | Article | PubMed - Weintraub WS and Diamond GA.
**Predicting cardiovascular events with coronary calcium scoring**.*N Engl J Med.*2008;**358**:1394-6. | Article | PubMed - Grundy SM, Bazzarre T, Cleeman J, D'Agostino RB, Sr., Hill M, Houston-Miller N, Kannel WB, Krauss R, Krumholz HM, Lauer RM, Ockene IS, Pasternak RC, Pearson T, Ridker PM and Wood D.
**Prevention Conference V: Beyond secondary prevention: identifying the high-risk patient for primary prevention: medical office assessment: Writing Group I**.*Circulation.*2000;**101**:E3-E11. | Article | PubMed - Califf RM, Armstrong PW, Carver JR, D'Agostino RB and Strauss WE.
**27th Bethesda Conference: matching the intensity of risk factor management with the hazard for coronary disease events. Task Force 5. Stratification of patients into high, medium and low risk subgroups for purposes of risk factor management**.*J Am Coll Cardiol.*1996;**27**:1007-19. | Article | PubMed - Wells S, Kerr A, Eadie S, Wiltshire C and Jackson R.
**'Your Heart Forecast': a new approach for describing and communicating cardiovascular risk?***Heart.*2010;**96**:708-13. | Article | PubMed - Williams BA, Dorn JM, LaMonte MJ, Donahue RP, Trevisan M, Leonard DA, Greene RS and Merhige ME.
**Evaluating the prognostic value of positron-emission tomography myocardial perfusion imaging using automated software to calculate perfusion defect size**.*Clin Cardiol.*2012;**35**:E14-21. | Article | PubMed - Vickers AJ, Cronin AM and Begg CB.
**One statistical test is sufficient for assessing new predictive markers**.*BMC Med Res Methodol.*2011;**11**:13. | Article | PubMed Abstract | PubMed Full Text - Vickers AJ, Elkin EB and Steyerberg E.
**Net reclassification improvement and decision theory**.*Stat Med.*2009;**28**:525-6; author reply 526-8. | Article | PubMed - Cook NR.
**Comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine**.*Stat Med.*2008;**27**:191-5. | Article | PubMed - Wilson PW, Smith SC, Jr., Blumenthal RS, Burke GL and Wong ND.
**34th Bethesda Conference: Task force #4--How do we select patients for atherosclerosis imaging?***J Am Coll Cardiol.*2003;**41**:1898-906. | Article | PubMed - Greenland S.
**The need for reorientation toward cost-effective prediction: comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine**.*Stat Med.*2008;**27**:199-206. | Article | PubMed - Bonow RO.
**Clinical practice. Should coronary calcium screening be used in cardiovascular prevention strategies?***N Engl J Med.*2009;**361**:990-7. | Article | PubMed

Volume 1

Williams BA. **Estimating the clinical effect of incorporating a new risk marker into therapeutic decisions via a risk reclassification strategy**. *J Med Stat Inform*. 2013; **1**:5. http://dx.doi.org/10.7243/2053-7662-1-5

View Metrics

Copyright © 2015 Herbert Publications Limited. All rights reserved.

Post Comment|View Comments