Abstract

Background: Surveys/questionnaires are common research tools to identify the most important barriers that physicians face to improve patient care. Typically, potential items are preselected by investigators and described in a question for participants to rank in order from most to least important. Problems may arise if not every item applies or if multiple items are equally ranked. The nature of this type of ranked data can complicate data analysis and the number of primary items ultimately selected is generally performed ad hoc. To overcome this issue, the question can be broken down into item-specific questions in Likert-style or visual analogue scale (VAS). These two scales are common in psychometric research, but the choice of appropriate statistical methods remains controversial based on the nature and scale of the data.

Methods: In this paper, we compare three question designs (ranking, Likert scale, and VAS) via simulations in order to identify primary items. We focus our investigation on the differences in designs by the scale of data and consider the VAS design as the gold standard as it produces data more informative than the rank and Likert data. We introduce a simulation-based method that accounts for correlation between ranks as well as ratings within subjects.

Results: The VAS design outperforms the Likert and ranking designs. The Likert design with 5 or 7 Likert items is not as compelling as the VAS design until the sample size is large. The ranking design tends to incorrectly identify primary items when the proportion of primary items is over 50%.

Conclusions: Overall, we conclude that the VAS design is the superior choice for identifying and prioritizing primary items. The Likert design can be improved by adding more items. The ranking design is adequate if the proportion of primary items is low.

Keywords: Likert scale, ranking scale, simulation study, questionnaire design, surveys, visual analogue scale, quality improvement