A design and evaluation framework for digital health interventions

: Digital health interventions (DHIs) have the potential to help the growing number of chronic disease patients better manage their everyday lives. However, guidelines for the systematic development of DHIs are still scarce. The current work has, therefore, the objective to propose a frameworkfor the designand evaluation of DHIs (DEDHI). The DEDHI framework is meant to support both researchers and practitioners alike from early conceptual DHI models to large-scale implementations of DHIs in the healthcare market.


Introduction
Over the last decades, the prevalence of chronic health problems, i. e. diseases, conditions, and syndromes that are continuing or occurring repeatedly for a long time, is steadily increasing.Chronic health problems include, for example, cardiovascular diseases, diabetes, chronic respiratory diseases (e. g., COPD or asthma), arthritis or certain types of cancer (e. g., multiple myeloma) [10,46,53].These health problems lead not only to a substantial decrease in the quality of life of those being affected [33,41,68] or loss in productivity [64] but represent also the most important economic challenge in developed countries with up to 86 percent of all healthcare expenditures [12,46,53].
In addition to current approaches to address this important problem, for example, through national chronic disease strategies and policies [89], the use of information technology to either monitor health conditions and behavior or to deliver health interventions is another promising approach to support the growing number of chronic patients in their everyday lives [1,53].In this article, we use the term digital health intervention (DHI), describing the action of intervening [66] with "tools and services that use information and communication technologies (ICTs) to improve prevention, diagnosis, treatment, monitoring and management of health and lifestyle" [26], which is closely related to the notion of mHealth, telemedicine, telecare and health IT [28].
In light of the mature history of evidence-based medicine with clear guidelines on how to develop and assess the effectiveness of biomedical or behavioral health interventions [70,82], up till now, guidelines for the systematic development and assessment of DHIs are still scarce and corresponding research has just started.For example, first evaluation criteria have been proposed during the last decade [8,9,18,27,60,80,83].However, this preliminary work lacks guidance to which degree and when to apply these criteria along the life cycle of DHIs [63,78].Particularly, it is essential to consider appropriate evaluation criteria not only during the conceptual and prototype phases of a DHI, but also with respect to long-term implementations in the health care market, so that a sustainable, effective and efficient use of DHIs can be achieved.The evaluation results would also be the foundation for trust-building certifications similar to energy efficiency labels of consumer products, which can be used by patients and health professionals alike to find the "right" DHIs.Moreover, barriers for implementation and scaling-up of DHIs remain [65] that intervention authors must be aware of and that need to be addressed during and after the development process.A successful DHI conclusively needs to consider both, the selection of suitable evaluation criteria and the overcoming of implementation barriers.Therefore, a match-making is deemed useful to assess which evaluation criteria need to be considered and which im-plementation barriers need to be addressed at which particular phase of a DHI life cycle.
The current work has therefore the objective to propose a framework for the iterative Design and Evaluation of DHIs (DEDHI) that describes a typical life cycle of a DHI and recommends relevant evaluation criteria and implementation barriers to be considered for each phase of this life cycle.
The DEDHI framework is meant to support both researchers and practitioners alike during the design and evaluation of various instantiations of DHIs, i. e., from conceptual models to large-scale implementations in the healthcare area.The scientific contribution lies in the alignment of research streams from different fields at the intersection of behavioral medicine (e. g., behavioral interventions), medical informatics (e. g., medical applications) and information systems research (e. g., barriers of health information systems, including aspects of technology acceptance).
The remainder of this article is structured as follows.Design and evaluation frameworks for health interventions and DHI life cycle models are presented in the next section which build the foundation of the proposed DEDHI framework.For this purpose, an extended version of the multiphase-optimization strategy (MOST) [19,20,21] is used as the guiding life cycle model.Then, a systematic literature review is described and a consolidated list of evaluation criteria for DHIs are presented.Afterwards and based on a previous literature review [65], a consolidated list of implementation barriers for DHIs are outlined.In the following main results section, the consolidated evaluation criteria and implementation barriers for DHIs are both mapped to the DEDHI framework.This mapping is conducted in a deductive manner by applying qualitative content analysis [55].Finally, the resulting DEDHI framework is discussed with recommendations for research and practice, and limitations.A summary and suggestions for future work conclude this article.

Design and evaluation frameworks for health interventions and DHI life cycle models
Various design and evaluation frameworks for health interventions have been proposed in the past.Examples of these frameworks are listed in Table 1.They range from guidelines for the development of public health interven-  [87] tions [87] and policies [24] at the population-level to behavioral health interventions [56] and DHIs [59] at the individual-level.A common shortcoming of these frameworks, however, lies in the lack of guidance with respect to evaluation criteria and implementation barriers along the different phases of a typical DHI life cycle.That is, appropriate guidance is missing from the conceptual model of a DHI to a product-grade DHI that is maintained in the longterm.In particular, none of these frameworks offers guidance on technology-related aspects (e. g. maturity, scalability or security) and there are only a few frameworks that consider the implementation phase explicitly [16,22].
To address these shortcomings, findings from DHI lifecycle models [13,39,47,77] can be used.These mod-els describe the phases that systems undergo while they evolve from a prototypical development to an operational product [84].For example, Broens et al. [13] proposed a four-layered life-cycle model.It distinguishes between the phases of prototypes, small-scale pilots, large-scale pilots, operational product and links specific determinants of successful DHI implementations to each of these phases.The generic Technology Readiness Level model also follows this structure but refines the initialization and prototype phases in a more granular way [54].
Against this background and in order to account for all relevant phases of DHI development and implementation, we propose an extended version of MOST [19,20,21] as the guiding life cycle model for DEDHI.It was selected, because (a) it describes the development of DHIs in a rigorous and iterative way with several design, optimization and evaluation steps and clearly defined optimization criteria, (b) it explicitly considers a novel class of personalized and promising health interventions, i. e., just-in-time adaptive interventions [61,62] and corresponding assessment methods such as micro-randomized trials [48] that heavily rely on the use of technology and, finally, (c) because it also focuses on behavioral health interventions at the individual-level which is relevant for chronic health problems [46,53].Due to the fact that MOST does not consider a phase after a DHI has been successfully evaluated in a randomized controlled trial, a corresponding implementation phase is added from both related design and evaluation frameworks from Table 1 [15,16,22] and a DHI life cycle model [13].Moreover, details on recommended maturity levels of DHI technology are also incorporated into this extended version of MOST from corresponding DHI life cycle models [13,54].
The proposed DEDHI framework, which is based upon this extended version of MOST, is shown in Table 6.This table also includes the consolidated evaluation criteria and implementation barriers for DHIs which are described in more detail in the following two sections.

Evaluation criteria for DHIs
A systematic literature review was conducted to identify evaluation criteria for DHIs.A recently published systematic review of quality criteria for mobile health applications [63] in combination with an explorative search in the PubMed and Google Scholar databases were used to identify appropriate search terms that revealed a significant amount of relevant search results.
The final set of search terms is listed in Table 2 and was applied as follows: (ID1 and ID2 and ID3 and ID4 in Title) and (ID1 and ID2 in Abstract) (note that ID refers to the search term ID listed in Table 2).The goal of the search strategy was to update and complement prior findings [63] due to the broader focus of the current work on DHIs which includes not only mobile health interventions but also web-based interventions and hybrid interventions in which also guidance by human health professionals are foreseen [51].The resulting search strategy therefore consisted of three approaches.First, a backward search was conducted with relevant work already identified by Nouri et al. [63] but with the broader focus on DHIs.Here, relevant articles were screened back to the year 2000, which can be determined as the start of systematic research on DHIs [3,4].Second, the work of Nouri et al. [63] was updated with the broader DHI focus and relevant work from December 2016 till May 2019.Third, the search strategy of Nouri et al. [63] was extended to socio-technical databases and journals, i. e., ACM Digital Library, IEEE Explore, and A-ranked and B-ranked digital health journals as listed in [75].An overview of the search strategy is outlined in Table 3.
A search result was included if the work was original, peer-reviewed, written in English, and described a tool with evaluation criteria for DHIs.Thus, systematic reviews of evaluation criteria were excluded but relevant work from these reviews was screened when published between January 2000 and May 2019.
The inclusion of relevant work was initially carried out by two authors of this article on the basis of title and abstract.In the event of uncertainty as to whether a particular work fulfilled the inclusion criteria, the entire text was read and, if necessary, a third co-author was consulted.The evaluation criteria with a corresponding def-  [55].In case of uncertainty, the two co-authors consulted each other and also included a third co-author to find a consensus.The systematic search led initially to 2616 journal articles and conference papers which were then screened step by step as outlined in Figure 1.
Overall, 331 evaluation criteria were then extracted from the resulting 36 records and consolidated into 13 categories.These categories are listed in Table 4 and accompanied by a description, references for further readings and the number of corresponding evaluation criteria.3.
Table 4: Consolidated categories of evaluation criteria for DHIs.Note: # EC = number of evaluation criteria (in % of all, i. e., 331); representative references are provided for further reading.

Name Description # EC
Ease of use The degree to which effort is required to take advantage of the DHI (e. g., using common interaction paradigms).[69,72,85] 87, 26.3 %

Content quality
The degree to which the content of a DHI is accurate, timely, complete, relevant, and consistent (e. g., real-time location-based pollen warnings for asthmatics).[45,79,81] 41, 12.4 %

Privacy & security
The degree to which the DHI considers legal requirements and aspects with respect to privacy and security aspects (e. g., a DHI is compliant with the General Data Protection Regulation).[17,49,74,88] 41, 12.4 %

Accountability
The degree to which information about the DHI is made explicit for usage decisions (e. g., details of the intervention author of a DHI are accessible).[23, 40,

Perceived benefit
The degree to which a person believes that using a DHI improves his or her health behavior/health condition (e. g., a believe that a DHI helps to increase physical activity).[17,73] 18, 5.4 %

Effectiveness
The degree to which the DHI contributes to the enhancement of an individual's health behavior/condition (e. g., significant reduction of fat mass).[17,67,73] 17, 5.1 %

Service quality
The extent to which support of a DHI is provided (e. g., a technical support line is made available).[49] 15, 4.5 %

Personalization
The degree to which the DHI adapts to the needs of an individual (e. g., the daily step goal of a DHI adapts to the capabilities of an individual).[14] 11, 3.3 %

Perceived enjoyment
The degree to which an individual believes that using a DHI is engaging (e. g., the use of game elements and level designs in a DHI).

Ethics
The degree to which the DHI addresses ethical aspects (e. g., the DHI was designed for individuals with various cultural backgrounds or disabilities).[88] 5, 1.5 %

Safety
The extent to which the usage of a DHI is safe with respect to side effects (e. g., interactions with a DHI are limited to account for addiction behavior).[88] 3, 0.9 % An overview of all selected articles, evaluation criteria and mapping of these criteria to the categories including examples is provided in [50].The results of the consolidated categories show that ease of use is by far the most dominant category, with 87 evaluation criteria.By contrast, evaluation criteria related to ethical and safety aspects of a DHI are so far quite neglected by the scientific community.Moreover, it can be observed that one fundamental aspect of evidence-based medicine and the primary objective of design and evaluation frameworks as outlined in Table 1, i. e., to assess the degree to which an intervention is effective, does not take over a prominent position with the eighth rank in Table 4. Finally, it can be noticed that both subjective evaluation criteria (e. g., perceived benefit of a DHI) and criteria measured objectively (e. g., adherence to a DHI) are listed among the resulting categories.

Implementation barriers for DHIs
A list of implementation barriers of DHIs was already identified in prior work by means of a systematic literature review of reviews [65].For the purpose of the current work, the 98 identified implementation barriers were summarized into inductive categories according to qualitative content analysis [55].Out of the 98 barriers, 106 assignments to categories could be made.This higher number is due to the fact, that some barriers are related to more than one category.An overview of the resulting categories of implementation barriers, their descriptions and numbers are shown in Table 5.

Mapping of evaluation criteria and implementation barriers along the DEDHI framework
The mapping of the evaluation criteria and implementation barriers for DHIs along the life cycle phases of the proposed DEDHI framework was conducted by means of a qualitative content analysis [55].The analysis was done by at least two scientists independently, whereby inconsistencies were resolved through discussion until consensus was reached.The resulting overview of the DEDHI framework, including the mapping of evaluation criteria and implementation barriers, is listed in Table 6.
For each phase of the DEDHI framework, the overall goal and corresponding design and evaluation tasks are outlined.These goals and tasks are adapted to the concept

Usability of technology
Effort needed to use the technology according to its purpose (e. g., lack of user centred design [43], complex use [34], problems with technical support [37]) 9, 8.5 %

Cost
Cost for start-up and maintenance [71], high cost of technology [37,43] 2, 1.9 % Culture Characteristics of the end user which are inherent in a group of users belonging together (e. g., language [71]) 2, 1.9 %

Interoperability
Ability of technology to be used with existing infrastructure (e. g., lack of interoperability [43]) 1, 0.9 %

Reimbursement
Missing reimbursement possibilities [30,36,43] 1, 0.9 % Responsibilities Different legal responsibilities (e. g., federalist medical law [43]) 1, 0.9 % Social interaction Interaction between patient and healthcare provider (e. g., limited personal connection, interaction and relationship [36,52,76]) 1, 0.9 % of DHIs from MOST [19,20,21] for the Phases 1, 2 and 3 and from related work on intervention design and life cycle models [15,16,22] for Phase 4 as outlined in Section 2. In addition, a brief description of the technical maturity of the DHI is provided to help intervention authors better understand the technical perspective.Moreover, relevant evaluation criteria and implementation barriers are provided for each phase of the DEDHI framework that are suggested to be addressed by intervention authors in order to create evidence-based DHIs that can be successfully implemented in the health care market.
While almost all criteria and barriers are only related to a single phase, some are related to two or all phases.For example, the two barrier categories funding and cost are related to all the phases as they represent start-up as well as maintenance cost and funding.Also, some individual characteristics (e. g., lack of trust in colleagues [43,71], lack of trust in politics [71], sticking to old fashioned modalities of care [43]) and negative associations of healthcare providers relate to more than one phase.First, they need to be considered within user-centered design processes in the preparation phase.Second, they can be addressed during the implementation phase by means of advertisement and awareness campaigns.Furthermore, usability also relates to more than one phase.However,

Preparation Phase (adapted from MOST [19, 20, 21]) Goals and tasks:
To define the conceptual and technological foundation of a DHI.That is, (a) to review existing justificatory knowledge for the development of a DHI (e. g., intervention need, best practices, meta reviews, market analysis), (b) to develop a conceptual model of the DHI outlining the outcomes and intervention components, (c) to conduct a feasibility and acceptability study to test novel DHI components, and finally, (d) to identify an optimization criterion that provides the best expected outcome within technical and health economic constraints.Technical maturity: Research prototype that provides basic functionality to assess the feasibility of the DHI Evaluation criteria: Ease of use, adherence, personalization, safety, privacy and security Implementation barriers: Social interaction, individual characteristics of end user, usability (e. g. user-centred design), expectations (privacy, confidentiality, security), negative associations, workforce, planning, funding (no funding), cost (start-up), standards, regulatory issues

Optimization Phase (adapted from MOST [19, 20, 21]) Goals and tasks:
To build an optimized DHI by selecting effective intervention components.That is, (a) to conduct optimization trials (e. g., a micro-randomized trial), and (b) to identify the best DHI configuration that meets the optimization criterion (e. g., a hybrid DHI that applies a chatbot that delivers health literacy information and a human lifestyle coach via a tele-medical online session).If the intervention components are not effective or the optimization criterion is not met, it is suggested to revisit the preparation phase and to make appropriate adaptations.Technical maturity: Elaborated research prototype that provides the full functionality according to the conceptual model to assess the health impact of DHI components Evaluation criteria: Effectiveness (individual components of the DHI), perceived benefit, content quality, personalization, perceived enjoyment, aesthetics, adherence, service quality, safety, privacy and security Implementation barriers: Social support, outcome expectations, usability, funding (for equipment), cost (of technology), integration 3. Evaluation Phase (adapted from MOST [19,20,21]) Goals and tasks: To confirm the effectiveness of an optimized DHI.That is, to conduct a randomized controlled trial to compare the optimized DHI with a control condition (e. g., treatment as usual).If the optimized DHI outperforms the control condition, then the optimized DHI may be implementation at large-scale.If this is not the case, it is suggested to revisit the preparation phase and to make appropriate adaptations.Technical maturity: Elaborated research prototype that provides the full functionality according to the conceptual model to assess the effectiveness of the DHI Evaluation criteria: Effectiveness, perceived benefit, adherence, personalization, service quality, safety, privacy and security, accountability Implementation barriers: Funding (no funding), cost (maintenance), guidelines, methodology  [15,16,22]) Goals and tasks: To implement and maintain the effective DHI.That is, (a) to develop a DHI product that can be implemented at large-scale in the healthcare market, (b) to monitor reach, impact and side effects to assess the long-term outreach and efficiency by performing a continuous outreach, impact and side effect assessment, and finally, (c) to update the DHI with respect to content (e. g., if new evidence is available; note: if the update represents a significant change of the causal chain in the conceptual model then it is recommended to reiterate through the preceding phases) and technology (e. g., if application programming interfaces of mobile operating systems are updated or extended, or if security threats make a technical update of the DHI necessary).Technical maturity: Elaborated research prototype that provides the full functionality according to the conceptual model to assess the health impact of individual intervention components Evaluation criteria: Adherence, personalization, perceived benefit, content quality, ethics, service quality, safety, privacy and security, accountability Implementation barriers: Individual resources of end users, expectations (missing accuracy of provided information), usability of technology (technical support, implementation problems), interoperability, human technical support, regional infrastructure, individual characteristics of healthcare providers, negative associations, accessibility, reimbursement, funding (long-term), cost (maintenance), culture different facets of the usability category relate to different DEDHI framework phases.
Finally, it must be noted, that some of the implementation barriers could not be aligned to the DEDHI framework as they cannot be overcome during the life cycle of DHIs but are instead related to framing conditions.This includes missing benefits, cooperation and responsibilities as well as characteristics of the disease involved which hinder the usage of DHIs in general.

Discussion
The DEDHI framework provides an overview of evaluation criteria and implementation barriers to be considered during the life cycle phases of DHIs.All criteria and almost all barriers could be matched to the four phases.However, all phases could be linked to different numbers of criteria and barriers, which underlines the importance of addressing both factors during the whole life cycle.Furthermore, it underlines the fit of the DEDHI framework regarding the purpose of informing DHI developers and evaluators step-wise about criteria and barriers to be considered.However, dependencies between criteria (e. g., lower relevance of costs whenever a DHI is easy to use and suffi-ciently helpful) were not considered in our work as they could not be identified by the literature review and content analysis itself.
The evaluation criteria and implementation barriers presented in this work originate from different countries and geographic regions, for example, the United States [30], Europe [36,69], Australia [69] or Africa [34,85].This shows the universality of the criteria and barriers and with it, also the universality of the DEDHI framework.
Moreover, it becomes obvious from the current work that the interdisciplinary field of Digital Health needs to integrate and consolidate perspectives and research findings from various fields such as behavioral medicine (e. g., the "active" ingredients of DHIs such as well-established behavior change techniques), computer science (e. g., machine learning algorithms embedded in DHIs that detect critical health conditions), software engineering (e. g., the rigorous design, implementation and test of DHIs) or information systems research (e. g., understanding the use and success factors of DHIs).That is, to better understand the development and evaluation of DHIs, it is crucial to broaden the scope and to account for related work at the intersection of the relevant disciplines involved.
Last but not least, no work comes without limitations which also applies to this one.First and foremost, the proposed DEDHI framework was developed purely in an inductive way based on content analysis techniques and existing justificatory knowledge.It was therefore not applied, validated and revised during the development and evaluation of DHIs in the field.Thus, empirical evidence that supports the utility of the DEDHI framework is not established yet.
Second, the current work considers findings from scientific outlets only and thus, incorporates country-specific regulatory frameworks only indirectly to the extent to which these regulations are covered by these outlets.That is, legal frameworks and prescriptions with respect to the life cycle phases will probably differ in detail and depend on the class of the (medical) DHIs in comparison to the more idealistic four phases of DEDHI.With the goal to accelerate the digital transformation of health care, for example, the German Ministry of Health proposes the implementation of easy to use and secure DHIs in a first phase before their effectiveness is assessed in a second step [86].This approach has the advantage, in particular for startup companies, that significant financial investments of up to several years (e.g., for optimization and evaluation trials) are not required up-front.Instead, in interdisciplinary collaborations with digital health (research or business) organizations, relevant stakeholders such as patient organizations, health insurance or pharmaceutical companies may take over a significant amount of these investments due to the early product character of DHIs.Another advantage is the primary focus on real-world trials compared to often artificial efficacy studies under controlled environments, for example, with highly selected participants or study nurses that are experienced with clinical trials [32].The major shortcoming of such an approach, however, is the fact that the burden of patients will be increased by offering DHIs that are (potentially) not effective at all.
Third, the proposed DEDHI framework does not make an explicit distinction between the goals and motivations of the various stakeholders interested in the design and evaluation of DHIs, such as research teams funded by national research foundations or commercial digital health companies which are dependent on payers such as health insurance organizations.While research teams may be primarily interested in the publication of novel digital coaching concepts and their impact on therapy adherence (here, the focus lies primarily on the preparation phase and optimization phase), the primary interest of commercial digital health companies may be to bring a new DHI as fast as possible into the healthcare market (here, the focus lies on the implementation phase).Implications for the documentation and testing of the DHIs may be very different in these cases.For example, the digital health company must establish well-documented software development processes at the very beginning of a new DHI project as they are hard regulatory requirements when the DHI is offered in the healthcare market.On the contrary, the very same regulatory requirements are not relevant for the research team.
And finally, the chosen methods include subjective procedures.Conducting literature searches and qualitative content analysis is limited by the terms and databases chosen, and by the subjectivity of the researchers involved.However, such bias was reduced as much as possible.For example, relevant databases were included for the searches and synonyms of search terms were tested for results.Furthermore, each methodological step was done by at least two authors independently and inconsistencies were resolved by discussion and consensus.

Conclusion and future work
Due to the lack of well-established design and assessment guidelines for digital health interventions (DHIs), the current work had the objective to propose a framework for the Design and Evaluation of DHIs (DEDHI).For this purpose, justificatory knowledge from the fields of behavioral medicine, medical informatics and information systems was reviewed.Overall, four life cycle phases of DHIs, 331 evaluation criteria and 98 implementation barriers were identified and consolidated.The resulting DEDHI framework is meant to support both researchers and practitioners alike during the various design and evaluation phases of DHIs.
Future work is advised to critically apply, reflect, validate, and revise the proposed framework with its components as the field of Digital Health is still in its nascent stage.Accordingly, it is recommended that experts from the fields of ethics, regulatory affairs, public health, medicine, computer science and information systems work closely together to pave the way for evidencebased DHIs.The latter would not only push the field of Digital Health forward but it will, first and foremost, help a significant number of individuals to better manage their chronic health problems in their everyday lifes.
Funding: This work was co-funded by Health Promotion Switzerland, and the European Social Fund and the Free State of Saxony (Grant no.100310385).

Figure 1 :
Figure 1: Overview of the screening process.Note: The #1-3 indicates the search strategy from Table3.

Table 1 :
Examples of design and evaluation frameworks for health interventions.

Table 2 :
Overview of search terms.

Table 3 :
Overview of the search strategy.

Table 5 :
Consolidated categories of implementation barriers for DHIs.Note: # IB = number of implementation barriers (in % of all, i. e. 106); representative references are provided for further reading.

Table 6 :
Overview of the DEDHI framework.