Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators.

Elizabeth McGill ORCID logo; Dalya Marks ORCID logo; Vanessa Er ORCID logo; Tarra Penney ORCID logo; Mark Petticrew ORCID logo; Matt Egan ORCID logo; (2020) Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators. PLoS medicine, 17 (11). e1003368-. ISSN 1549-1277 DOI: 10.1371/journal.pmed.1003368
Copy

BACKGROUND: Public health evaluation methods have been criticized for being overly reductionist and failing to generate suitable evidence for public health decision-making. A "complex systems approach" has been advocated to account for real world complexity. Qualitative methods may be well suited to understanding change in complex social environments, but guidance on applying a complex systems approach to inform qualitative research remains limited and underdeveloped. This systematic review aims to analyze published examples of process evaluations that utilize qualitative methods that involve a complex systems perspective and proposes a framework for qualitative complex system process evaluations. METHODS AND FINDINGS: We conducted a systematic search to identify complex system process evaluations that involve qualitative methods by searching electronic databases from January 1, 2014-September 30, 2019 (Scopus, MEDLINE, Web of Science), citation searching, and expert consultations. Process evaluations were included if they self-identified as taking a systems- or complexity-oriented approach, integrated qualitative methods, reported empirical findings, and evaluated public health interventions. Two reviewers independently assessed each study to identify concepts associated with the systems thinking and complexity science traditions. Twenty-one unique studies were identified evaluating a wide range of public health interventions in, for example, urban planning, sexual health, violence prevention, substance use, and community transformation. Evaluations were conducted in settings such as schools, workplaces, and neighborhoods in 13 different countries (9 high-income and 4 middle-income). All reported some utilization of complex systems concepts in the analysis of qualitative data. In 14 evaluations, the consideration of complex systems influenced intervention design, evaluation planning, or fieldwork. The identified studies used systems concepts to depict and describe a system at one point in time. Only 4 evaluations explicitly utilized a range of complexity concepts to assess changes within the system resulting from, or co-occurring with, intervention implementation over time. Limitations to our approach are including only English-language papers, reliance on study authors reporting their utilization of complex systems concepts, and subjective judgment from the reviewers relating to which concepts featured in each study. CONCLUSION: This study found no consensus on what bringing a complex systems perspective to public health process evaluations with qualitative methods looks like in practice and that many studies of this nature describe static systems at a single time point. We suggest future studies use a 2-phase framework for qualitative process evaluations that seek to assess changes over time from a complex systems perspective. The first phase involves producing a description of the system and identifying hypotheses about how the system may change in response to the intervention. The second phase involves following the pathway of emergent findings in an adaptive evaluation approach.


picture_as_pdf
McGill et al 2020.pdf
subject
Published Version
Available under Creative Commons: NC-ND 3.0

View Download

Atom BibTeX OpenURL ContextObject in Span Multiline CSV OpenURL ContextObject Dublin Core Dublin Core MPEG-21 DIDL EndNote HTML Citation JSON MARC (ASCII) MARC (ISO 2709) METS MODS RDF+N3 RDF+N-Triples RDF+XML RIOXX2 XML Reference Manager Refer Simple Metadata ASCII Citation EP3 XML
Export

Downloads