
Chun Lok K. Li
Correspondence: Chun Lok K. Li krislichunlok@hotmail.com
Author Affiliations
Melbourne Institute of Applied Economic and Social Research, The University of Melbourne, Victoria, Australia.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Background: Policymakers in many countries are confronted with the question of whether, and how much, resources should be committed to improving hospital quality. This is because there is little consensus about the relationship between hospital quality and cost, despite the extensive existing literature. This makes it difficult for policyholders to commit resources to make improvements. The diversity of methods used also makes cross-comparison difficult. To address this, two specific methodological issues that are commonly observed will be investigated. The first is the choice of metric used for assessing hospital quality. The second is the way the distribution of the measure is specified.
Methods: An empirical example will be setup that resembles a typical study on this research topic. The purpose is to demonstrate that a change in the metric used for measuring hospital quality, or a change in its distributional assumptions, will lead to a different result, even when the same data is used. For simplicity, the two most general metrics of patient mortality and unplanned readmissions will be used. The measuring statistic is the odds ratio to allow for patient risk-adjustment. A bootstrap-adjusted regression modified from Lindley and Smith (1972) will be used to account for the distribution of the quality statistic. Hospital cost data is derived from financial statements on a subset of Victorian public hospitals from 2002/03 to 2004/05. Patient data is sourced from encoded records in the Victorian Admitted Episodes Database (VAED) for the six years from 1999/00 to 2004/05, which are used for patient risk-adjustment. Hospitals and patients are anonymously linked.
Results: The relationship between quality and cost is negative at the 95% significance levels when the bootstrap adjustment is applied, for both adjusted mortality and readmissions. When the adjustment was removed, the relationship became positive with mortality and close to zero for readmissions, both with much wider confidence intervals. A hypothetical static comparison exercise resulted in a difference in estimated costs of around several million AUD a year in operating costs of public hospitals in the state of Victoria.
Conclusion: When determining the effectiveness of hospital policies, more than one operative measure of quality should be used as a robustness check. The dispersion of quality measures must be explicitly accounted for. MeSH codes: N03.219.262 (Hospital Economics); N05.300.375.500 (Costs, Hospital); N02.278.421.510 (Hospitals, Public); N04.452.871.715.800 (Risk Adjustment); E02.760.400.620 (Patient Readmission); E05.318.308.985.550.400 (Hospital Mortality); E05.318.740.750 (Regression Analysis).
Keywords: Hospital economics, risk adjustment, costs, hospital, regression analysis, hospitals, public, hospital mortality
The relationship between hospital quality and cost is an established research question. For example, in the US, hospital dummy variables were used as a measure of quality in a study that investigated the revealed preference of pneumonia patients in Los Angeles [1]. In Hong Kong, a medical study used the "30-day odds-ratio" to a selection of acute illnesses that are representative of the dataset [2]. And in Australia, the accuracy of adverse incidence rates and its impact on measuring hospital performance was investigated, with a strong emphasis on patient risk-adjustment [3]. The interested reader can find many more.
However, despite the abundance of existing studies investigating this research question, there is still a lack of a clear consensus as to whether the relationship between hospital cost and quality is positive, negative or not related. Without a representative conclusion, there is no benchmark to use for empirically validating new theories claiming to explain the phenomenon, and policymakers in many countries are still unsure whether suggested policies to improve hospitals will be effective. Some of these differences can be attributed to differences in data used, in which case a diversity of results could simply reflect a strong geographical influence. But the problem is that, given the differences in measures used for assessing hospital quality statistics, and the strong assumptions used for the specification of said measures, it is difficult to tell either way. A wide variety of metrics can be used to measure patient out- comes [4]. Some examples that are directly related to hospital studies are different types of ambulatory care [5] or various hospital-acquired infections. There is little consensus on which type of incidence to count, except that more than one measure should be used if the data and research question allow [6].
As for proper specification of hospital quality statistics, data limitations are the most commonly cited issue. The method of first resort is usually instrumental variables, solving the data shortage issue and reducing endogeneity at the same time. But finding a good instrument is often challenging because it needs to be closely related to the variable in question, and at the same time uncorrelated with the other variables. Some well-known examples include the patient's geographical distance from the nearest hospital [7] or even time differences [8]. Most studies do not have such clean solutions.
This paper provides an empirical reference to the literature that highlights the necessity of testing results using more than one quality measure, and the need to capture the distribution of said measures. By using the same data and only varying the estimation method in these two ways, any differences in results will provide further support that future studies will also need to introduce similar variations. The aim is to expedite progress towards this research question by enabling cross-comparison between studies, thereby reducing the uncertainty facing policymakers in this regard. The next section will explain the empirical setup.
This section is organised as follows. A model for capturing the distribution of a hospital-level Odds Ratio (OR) will be derived. An illustrative example, that of a typical empirical analysis between hospital quality and operational cost, will be described. The estimation strategy will be explained.
Modelling the hospital odds ratio
The OR is a generic measure that can be applied to most hospital
incidents. This study will measure hospital quality using
two incidence types, mortality and unplanned readmissions
(henceforth referred to as 'readmissions'). Mortality is clearly
defined, while one readmissions count is defined in the data
as an unplanned revisit within 28 days from the same patient
for the same problem.
There are H hospitals in the data, indexed h = 1,...,H. The patients that each hospital admits are indexed p = 1,...,Ph. The mortality status of each patient is modeled as a binary variable 0 or 1, denoted as Dhp with probability of occurrence E(Dhp). The number of previous readmissions of each patient is a count variable Rhp with expected value E(Rhp).
Each hospital h has a specific number of mortality incidents
per year Dh and readmissions Rh. Their estimates, denoted as
and
respectively, are defined as follows:
The number of hospital incidents Dh and Rh are random
variables, with corresponding expected values of E(Dh) and
E(Rh). They are computed as follows:
The OR of these two incidence types, written as ZD and ZR is
then the ratio of observed to expected number of incidents:
We want to capture the stochastic processes of ZDh and ZRh. By construction, they contain four random variables Dh Rh E(Dh) and E(Rh) each following its own distribution.
Dh is the sum of binary variables Dhp it follows a binomial distribution with probability of incidence equal to Dh/Ph.
Rh is the sum of count variables Rhp which follows the Poisson distribution with expected count rate Rh.
E(Dhp) is estimated using regression because probabilities
are not directly observed. Let δhp be the vector of dependent
variables associated with each patient p in hospital h. Each
element in that vector has a corresponding parameter
estimate that captures their marginal effect on incidence
probability. Denote the vector of parameters for mortality
using λD. Mortality is a binary dependent variable, estimated
using Logistic regression:
E(Rhp) is a count variable, estimated using Poisson regression
[9]. Let λR be the vector of parameters. The estimation is as
follows:
Both Logistic and Poisson regressions are estimated using standard maximum-likelihood estimation (MLE). This study used STATA v12; it is also available in most major off-the-shelf statistical software.
Computing point estimates of ZDh and ZRh is straight forward. However, when they are included in a cost function for estimation, its error structure needs to be accounted for in order to reduce unobserved bias. Standard computed regressor methods exist for this purpose, but they only work for variables with Gaussian distributions or some other simple functions. This is not the case for ZDh and ZRh where each is formed as the ratio of two random variables. Their algebraic structure does not lend itself to clean, analytical solutions either.
Approximating their distributions using parametric simulation can get around this problem. Let εD and εR be the vector of standard errors for λD and λR respectively. Index the patient variables used for risk-adjustment from u = 1,...,U and, accordingly, each element in the given vectors.
The distributions of ZDh and ZRh can be simulated based on
the regression assumption that each estimated parameter
follows the normal distribution, given as follows:
By generating a large number of simulated risk-adjustment parameters and re-computing ZDh and ZRh we produce an EDF of the estimated quality of each hospital. For this study, parametric simulation is repeated for a total of N=1000 values for each ZDh and ZRh. Results are denoted as ZDh1 and ZDhN and ZRh1 to ZRhN respectively. The use of these simulated quality values are explained later.
The empirical setting
The relationship between hospital quality and cost is an
established research topic without a consensus opinion. The
focus here is the differences in estimation results when the
distribution of the quality variable is controlled for. To do this,
we construct a hospital cost function with input prices, output
units and risk-adjusted quality measures. Throughout this
study, individual hospitals cannot be identified and summary
statistics are presented only in aggregate.
Input
Hospital input prices are available for public hospitals in
Victoria, Australia from the years 2002/03 to 2004/05 (Table 1).
They are extracted from financial accounting statements, with
years chosen based on similarity in reporting requirements.
To be included in this study, a hospital must have enough
entries in their financial statement to compute total expenses,
capital prices (K), labour prices (L), and materials prices (M).
Table 1 : Selected hospitals in the VAED, 2002/03 to 2004/05: an overview.
Output
The unit of output is the number of patient-days per episode.
They are available in the Victorian Admitted Episodes database
(VAED) for the six years from 1999/00 to 2004/05 (Table 2). It
contains basic information such as demographics, disease diagnostics and how the patient was discharged. All six years
of VAED patient data are used for patient risk-adjustment,
while the last three are included into the cost function through
data-linkage with input prices.
Table 2 : Summary of patient mortality rates in VAED by year and Major Diagnostic Categories.
Risk-adjusted quality
Regression variables included for risk-adjustment include:
age, gender [10], and the ICD-10-AM Charlson comorbidity
index [11]. Additionally, the risk-factors for patients in each
of the 23 Major Diagnostic Categories (MDC) as listed in the
VEAD are separately estimated. The advantage over the more
common fixed-effects approach is it allows the same variable
to exert a different marginal effect depending on the MCD.
Note that, since patient data for this study begins at the year 2002/03 and ends at the year 2004/05, the readmissions rate for episodes that occurred near the first and last 28 days of the observation period will be under-represented.
Estimation Strategy
The hospital cost function used in this study contains the total
operating cost of each hospital in a given financial year as the
subject variable and all the explanatory factors contributing
to the total cost,
where w is the vector of input prices, y is the number of patient-days and z are the risk-adjusted mortality and readmissions measures.
We approximate the hospital cost structure using a translog
function [12]. Let i and j be indices for input prices. The reduced
form is given as follows:
The cost function is estimated using seemingly-unrelated regressions (SUR) [13], which improves estimation accuracy by imposing linear homogeneity and reducing standard error. It is now available in most commercial software; see Christensen et al., (1973) for the technical derivations.
The statistical uncertainty in the quality measures ZD and ZR also needs to be accounted for. We do this using a bootstrap regression [14]. Recall that each hospital has two observed quality measures ZDh and ZRh, each associated with a set of N=1000 simulated quality values ZDhn and ZRhn. If each of the observed hospital quality measures is replaced with a simulated one, then the resulting parameter estimates will be different as well.
For each bootstrap n = 1,...,N, denote the resulting parameters
as
and the corresponding covariance matrix
.
These bootstrap-estimates form the distribution of the
parameter estimates. The next step is to compute the weightedaverage
of the parameters, written as
. The method used here
is similar in concept to Lindley and Smith [15], where each
successive bootstrap is treated as additional information added
to the prior. This gives us the weighted-average estimate the
covariance matrix, estimated as given:
The weighted-average estimates of β follows:
Point estimates from both the unadjusted and adjusted versions will be presented for comparison.
Parameter estimates of the cost function estimated with
adjustments for the distribution of the quality measure
are listed in Table 3. The risk-adjusted mortality and readmissions
parameters
and
are both negative and statisticallysignificant
at the 95% level. It means that, given all else
equal, lowering incidence rates will lead to increased cost.
Table 3 : Parameter estimates by bootstrap regression.
When the same cost function is estimated without the
adjustment as in Table 4,
becomes positive and
becomes
close to zero, both with higher standard errors such that they
are no longer statistically significant. This suggests that the
variability of the quality indicators are considerable and that
the resulting biases from ignoring it during regression are large.
Table 4 : Parameter estimates without bootstrapping.
A comparison between estimates generated with and without adjustment serves to highlight the importance of controlling for the statistical properties of hospital quality. However, the practical implications of these differences are not easily understood just by inspecting these numbers in isolation.
To put these differences in policy context, a static comparison
exercise can be used to evaluate the magnitude of the possible
financial miscalculations from an adjustment in cost function
estimation methods. The setup is a hypothetical policy
intervention whereby hospitals below a certain industry
standard for patient outcome are given resources to bring
their operations up to a minimum target level:
Here the symbols
and
represent a quality level that
hospitals need to raise themselves up to. For this exercise, we
choose 1.0 (parity) and 0.5 (hypothetical). Hospitals already
operating at or above those levels are left alone. The quantity
of interest is the difference in total operating cost between
hospitals operating at the observed versus hypothetical
levels of quality.
Dollar values for the static comparison results are displayed in Table 5. In the case of the sampled Victorian hospitals, the magnitude is around 1 million AUD per year for the 1.0 "industry standard" scenario. This number increases to around 10 million for the 0.5 "major improvement" scenario. The difference in both cases is a change in sign.
Table 5 : Static comparison: estimated cost of intervention.
Note also that mortality and unplanned readmissions exert a different effect on hospital expenses. Both are measures of hospital quality, suggesting that the financial impact of quality improvements does depend on which variables are used to measure quality when evaluating policy options for improving hospital quality.
When the cost function was estimated with an adjustment to the distribution of the quality variables, there is a negative and statistically-significant relationship between risk-adjusted incidence rate and hospital cost. In contrast, when the adjust ment is removed, the correlation became positive and not significant. This comparison provides a reference to the literature supporting the notion that the distribution of quality measures needs to be captured.
As an example, consider an authoritative study done in Japan on its hospital spending and quality [16]. Their main research question is why Japan is able to have superior quality for lower costs as compared to other OECDs. The core analysis consists of a log-log constant elasticity function where hospital quality is the dependent variable and hospital characteristics are the explanatory variables. One of their central measures of quality that is similar to OR, namely the Standardised Mortality Ratio, is an estimated measure with standard errors that are not accounted for. This may distort the evaluation of the quality of individual hospitals in the analysis, possibly resulting in less accurate parameters for determinants of quality.
We have also reinforced the importance of the choice of variable used to measure hospital quality by comparing between two separate variables, mortality and unplanned readmissions. The two variables had different marginal effects on hospital cost, even under controlled experiment settings, where the same dataset, cost function and estimation strategy were used.
The weakness of aggregating quality measures into a single index can be further illustrated using a study on US public hospitals in several states [17]. Their main message is that the negative impact of poor quality on hospital efficiency is underestimated in existing literature. The basis for their argument is that they found a stronger correlation using a multiple-indicator measure as compared to a more popular single-indicator measure, the Acute Myocardial Infraction Indicator (AMI). However, since their analysis did not include estimating the impact of individual components in the aggregated measure, there is a chance that their result was driven by a couple of outlier components rather than a general underlying relationship.
We recommend that the stochastic properties of quality measures should be explicitly captured for future empirical studies. Also, as demonstrated in this study, more than one quality measure must be used for any assessment. We used Australian hospitals as an example but a similar notion applies to hospitals from other countries. Additionally, this study's approach can be used as a starting basis for exploring the stochastic properties of other commonly-used quality measures. It can also be used to explore other, more effective methods for capturing the multiple dimensions of quality in an empirical setting. Both are ongoing research areas.
The author declares that he has no competing interests.
The author acknowledges the Melbourne Institute of Applied Economic and Social Research for providing the data and research funding for this study.
Editor: Abdulbari Bener, Weill Cornell Medical College, Qatar.
EIC: Jimmy Efird, East Carolina University, USA.
Received: 16-Mar-2014 Final Revised: 29-Apr-2014
Accepted: 05-May-2014 Published: 19-May-2014
Li CLK. Hospital quality measure specification and their estimated relationship with hospital cost. J Med Stat Inform. 2014; 2:4. http://dx.doi.org/10.7243/2053-7662-2-4
Copyright © 2015 Herbert Publications Limited. All rights reserved.