Evaluating complex interventions using routinely collected data: Methods to improve the validity of randomised controlled trials and observational studies

This thesis addresses the evaluation of complex interventions using routinely collected data, specifically the internal validity of observational studies and the generalisability of Randomised Controlled Trials (RCTs). Following a literature review, this thesis has four main objectives: to estimate the effect of telephone health coaching on hospital utilisation in an observational study; to assess optimal choices of control area in observational studies; to estimate the effect of telehealth within a large RCT; and to develop methods to assess aspects of the generalisability of RCTs empirically. The first paper compares health-coached patients with matched controls. Controls were selected from areas of England that were first matched to the characteristics of the intervention area. Health coaching did not reduce hospital admissions in this study. A second paper uses simulations to assess the relative bias and statistical precision in the treatment effects estimated under alternative approaches to selecting control areas. Lower bias is reported when using local controls than when selecting controls from matched areas, except when there is little unexplained area-level variation in outcomes, when the opposite is true. The third paper reports that, in the RCT, telehealth patients had fewer hospital admissions than controls, but admissions increased unexpectedly among controls after recruitment, leading to concerns about generalisability. Placebo tests find that control patients in the RCT experienced more admissions than matched non-participants receiving usual care. To address the concern that the control group did not receive ‘usual care’, sensitivity analyses are presented that contrast outcomes between the telehealth patients in the RCT and matched non-participants. In this comparison, telehealth is associated with a trend towards more admissions than usual care. The thesis concludes that careful control matching and placebo tests can address important aspects of the validity of observational studies and RCTs, but that further development of evaluation methods is warranted.