User engagement in mobile apps for people with schizophrenia: A scoping review
User engagement in mobile apps for people with schizophrenia: A scoping review Raquel Simões de Almeida
Introduction
The growing prevalence of digital technologies in healthcare has created great interest in mHealth in the Psychiatry field. With the constant development of smartphones, mobile applications (apps) have become important tools for assisting with the diagnosis, educating, monitoring, treating, and encouraging people to manage their mental health problems. Besides providing a unique opportunity to expand the availability of mental health assessments and interventions, apps could enhance the quality of the services delivered. This is important since the contemporary paradigm determines that people with mental health problems, especially those with schizophrenia, could have access to digital training and take more responsibility for their own care collaborating with clinicians to attain their personally defined and valued goals, using their self-determination, opposing the high chronicity, disability, and the burden continuously associated with this illness (1).
Regarding technology use, several studies found that mobile phone ownership among patients with schizophrenia is high, around 80%, and that the majority owned a smartphone (1–5). Younger patients have fewer negative symptoms and better psychosocial functionality and were likely to use technological devices more (6). Moreover, among people diagnosed with schizophrenia, the willingness to use smartphone apps is associated not only with age, but also with education, income, and device type (7). Also, some mental health practitioners are apprehensive that psychotic symptoms such as paranoid delusions may limit the feasibility of mobile health interventions. However, several studies have shown that mobile interventions are a viable strategy for people with psychosis, and they express their satisfaction with these interventions, finding them useful, beneficial, and easy to use, not triggering a worsening of symptoms (8, 9). Still, patients with more adverse symptoms or more severe intellectual deficits will struggle to use apps and could benefit more with other interventions.
However, high interest in mobile apps for mental health does not automatically mean high involvement in using them over time which affects the effectiveness of mobile interventions (10–12). Mental health apps have high dropout rates and a very low user engagement over time, not only in clinical trials but also in the real-world (13).
Some authors argue that this situation is due to negative or positive symptomatology or cognitive disfunction (3, 14), however it could also be cause by a poor user experience and usability. A study from Alqahtani and Orji (15) said that poor usability, lack of a content variety and personalization, lack of customer service, and security and privacy issues are the most common reason for abandoning mental health apps. Other authors claim that user experience does not predict sustained engagement with mental health apps and that it is necessary to define what sustained engagement means in this scope, what leads to it, and how to create products that achieve it (16). There are several engagement definitions (17) and defining meaningful engagement for digital mental health interventions remains a challenge as well as comparing engagement across studies or tools (18). A good example of engagement definition within digital interventions can be translated as “the extent (e.g., amount, frequency, duration, depth) of usage and a subjective experience characterized by attention, interest and affect” (19). Being a multidimensional concept, critical for mHealth effectiveness, it is essential to have a clear understanding of which user engagement indicators to report, namely “usability,” “user satisfaction”, “acceptability”, and “feasibility”. Moreover, there are subjective and objective criteria used to assess user engagement indicators, and behavioral measures tend to focus more on product metrics rather than in the recovery process success.
The definition of success regarding user engagement, means different things to different audiences. For instance, for a patient experiencing acute symptoms, the simple act of logging in the app and asking for support can be seen as a successful experience. For other users that may benefit from regular and repeated interaction with an app, success may be defined by the number and the quality of interactions between the user and the app. Both scenarios have very different definitions of success (20).
There are several models and theories that study engagement strategies. One of these models is the Persuasive System Design (PSD), a framework that defines four different mechanisms to increase user engagement by (1) focus on the central purpose of the app, (2) promoting user-app interactions, (3) foster social relationships between users and other profiles, and (4) increasing app credibility (21). Some mHealth apps use game-design elements to foment and maintain health-behavior outcomes. Although evidence suggests that gamification can boost health-behavioral outcomes, the results are not consistent and does not guarantee long-term use and adherence to mHealth (22).
Some authors argue that strategies complementary to the app's core feature such as reminders, feedback, coaching and peer support, tend to increase user engagement. Still, the absence of a common criteria that can measure engagement and usability across studies, makes it harder to distinguish which of the studies around the subject really bring some improvement on user engagement, regardless of the common claims of success by most of these works (18).
Despite the promise made by some mHealth apps for people with schizophrenia that enable them to self-manage their illness, adoption rates are relatively low and long-term user engagement is still an issue. Thus, in this scoping review we aim to identify:
(1) What strategies are applied to improve user engagement with mobile apps for schizophrenia self-management?
(2) What factors result in better user engagement for people with schizophrenia?
(3) How user engagement is being measured?
Materials and methods
This scoping review was conducted in accordance with the five-stage process outlined by Arksey and O'Malley (23) identifying the research question, identifying relevant studies, study selection, charting the data, and collating, summarizing, and reporting the results and is reported in line with PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines (24) reporting guidance. The protocol was made available to the Open Science Framework (https://osf.io/awh3z/) in May 2022.
Search strategy
Systematic search queries of Web of Science, PubMed, Academic Search Complete and IEEE Explore were used to identify references published or available online in the last 10 years, to avoid discussing mobile apps that are potentially out of date. This search was done from April to June 2022 and all types of primary peer-reviewed research papers were considered. A sample search strategy comprised the terms: (“schizo*” OR “psychosis” OR “psychotic disorder*”) AND (“self-management” OR “self-help”) AND (“mhealth” OR “app” OR “smartphone”). The chosen keywords were selected based on the literature and the specific aim of our study.
Eligibility criteria
After the records obtained were imported onto Rayyan (25), which is an online platform designed for multiple reviewers to work on systematic reviews, titles and abstracts were reviewed for eligibility. Reviewers are kept blind to each other's decisions, and can mark records as “include,” “exclude,” or “maybe” and can also mark exclusion reasons or add notes. This process was used to determine which records would be brought forward to full text review.
The PCC framework (Population or Participants)/Concept/Context) was used to guide study selection and to align the eligibility criteria with the research questions. Accordingly, to meet the eligibility criteria, the included papers had to (1) address a mobile app (programs or software applications designed to run on a mobile device such as a smartphone), (2) be designed for patients with schizophrenia spectrum disorders, (3) to manage their illness (including interventions such as relapse prevention, adherence to medications and/or treatment, psychoeducation, symptom monitoring) and (4) be published in English. Only apps designed specifically for people with schizophrenia spectrum disorders will be included, not general mental health apps used by those with these conditions since the purpose is to study how the applications for this population should be designed considering their characteristics. These papers should report on some aspects of user experience, using specific concepts, namely “usability,” “user satisfaction”, “acceptability”, and “feasibility”. There were no restrictions on characteristics pertaining to participants (e.g., age, ethnicity, gender), population (e.g., adult, youth) or setting (e.g., clinical, community).
The exclusion criteria were as follows:
- Studies without primary data, including reviews, commentaries, and study protocols.
- Studies that did not report engagement strategies in the intervention design.
Study selection
Study selection was done in two steps. First, after duplicates were removed, the titles and abstracts of all retrieved articles were screened for eligibility by two authors (RSA and AM). Next, the full text of all remaining publications was checked for inclusion by the same authors. Disagreements on the inclusion or exclusion of publications were discussed until agreement was reached. The average percentage of agreement between authors was approximately 90%. Consistent with the PRISMA-ScR guidelines and the framework proposed by Arksey and O’Malley (23), a quality appraisal was not conducted.
Data extraction
Data from eligible papers were extracted by one author (RSA) and checked for accuracy by the other author (AM). A data-charting form was developed and calibrated by the team. Data items that were extracted from each included study were authors names, country of origin, year of publication, study design and purpose, app name, type of engagement strategy, factors influencing engagement, used measures, and main findings.
Results
The search yielded 232 unique titles after duplicates were removed. After title and abstract analysis, 43 papers were assessed for eligibility and full text screening. In total, 27 papers published from 2012 up until 2022 were included (Figure 1).
Figure 1. Selection of studies - PRISMA flowchart.
Study characteristics
Most of the publications emanated from the USA (n = 10), nine publications originated from UK, two from Canada, two from Australia and the others from other European countries (Denmark, Poland, Portugal, The Netherlands, Spain) (Figure 2). Mixed methods (n = 12) and quantitative studies (n = 11) were the most common data collection methods used, accounting for 82% (23/28) of all studies. Most samples used in these studies were relatively small ranging from 5 to 361 participants. Study duration (which in this article translates to the time app was used/tested) ranged from 6 days to 14 months.
Figure 2. Number of included articles by year of publication and continent of authors.
Regarding included studies, some of them analyzed data from the same apps, however, we included them since the sample and the information retrieved were different. Apps like FOCUS (26–29), SMARTapp (30), MACS (31), weCope (32), My Journey 3 (33, 34), and App4Independence (35) offers illness management strategies in general. Others are more focused on symptom monitoring, namely Crosscheck (36), Actissist (37), ClinTouch (38), ExPRESS (39), Ginger.io (40), ReMindCare (41), and MindFrame (42). Other mobile applications are supported in CBT (SlowMo, 43; TechCareApp, 44; movisensXS, 45; PRIME, 46), or work as an agenda (MONEO, 47) or target specific domains such as social skills (MASS, 48), sleep (ExpiWell, 49), medication adherence (MedActive, 50), or delivers positive psychology content (+Connect, 51). Included studies are described in Table 1.
Table 1. Characteristics of included studies.
Strategies to improve user engagement
Not all the studies included in the scoping review focused engagement concretely, but all of them address this issue in some way (Table 2). As all this data is the result of studies carried out, downloading the apps is something already integrated into the process. What is important to comprehend is how to enhance all the ways users interact with the apps, from buttons they click to time spent with the app open.
Table 2. User engagement indicators.
Almost every app uses push notifications or prompts, except the one from Simões de Almeida et al. (32), and the frequency of notifications are usually at least daily. Personalization and customization were also a strategy used in ten studies (30, 35, 37, 42, 45, 46, 48, 50–52) and could be associated to the design or to the content as well. Another common strategy presented in six studies is goal setting, i.e., present a goal-oriented process (32, 33, 35, 46, 48, 50). Some studies relied on game-like features (e.g., points, challenges, badges) (43, 46, 51, 52) or the use of different multimedia formats (e.g., videos, music, images) (29, 37, 43, 44, 47, 48, 52) to encourage participant engagement. Benefits to long-term engagement may occur using real-time data monitoring and passive monitoring. Peer-to-peer communication (34, 35, 46), access to professionals (32, 34, 41), reliability of content (52), and the quality of feedback received (50, 52), were also important strategies used to improve user engagement in these included studies.
Factors that influence user engagement
There are three factors that may influence user engagement that constantly arise from these studies: age, illness severity, and technical support. Other demographic variables may can play a role in engagement namely gender (26, 35), education level (26), and socioeconomic status (55). Issues about data privacy and security and integration with other medical record systems were also described (29, 32, 34, 42, 44, 51). There are also some putative factors which could influence engagement as well, such as whether patients were involved in the app design process, incentives for participation, etc. (32, 40, 42, 43, 52). Most of these apps included at some stage user involvement in the development process and in most of the studies, participants were not monetarily incentivized to engage in interventions but were compensated for completing assessments. Any engagement strategies used should consider the need for initial training for using the app (including training sessions, smooth onboarding, possibility to ask questions and contact a support team to manage any technical difficulties) since some users could lack digital literacy/fluency (26–28, 30, 31, 33, 36, 39, 46, 48, 55). The fact that the use of the app fits well into the user's routine and that this use is encouraged by the clinician (or based on theoretical foundations and evidence) are also factors that influence engagement (32, 33, 50, 52, 55). Moreover, it is important to consider that user motivation to use mHealth may decline over time.
Measures for user engagement
Included studies evaluate user engagement indicators with different methodologies, from the criteria (such as subjective ratings and objective data) to the means of assessment (such as a survey, interview, or usage data). Besides metrics collected through the app (including overall time used, time and frequency of using specific functions, frequency of app refresh, daily check-in responses, …) (26, 28, 30, 32, 35–37, 39, 41, 43, 44, 46–48, 51), most of studies used questionnaires and semi-structured interviews created to assess satisfaction, acceptability, feasibility, and usability (27, 29, 30, 32–35, 37, 39–43, 48, 50, 55). Some studies used validated instruments such as System Usability Scale (27, 31), Usefulness, Satisfaction and Ease Questionnaire (27, 31) or Service Engagement Scale (33, 34).
Discussion
The findings of this scoping review have several implications for future research and give indications for designing and implementing mobile mental health applications.
First, it is important to note that in this context, engagement is measured by the frequency the user interacts with the app and by the impact that the use of the app has on the user's health (17).
Some human–computer interaction researchers define user engagement as catching and keeping the attention and interest of technology users and the cognitive, emotional, and temporal investment made by users (10), which should be adapted to this field, since people with schizophrenia have fluctuating needs. In this perspective, disengagement may be the visible result of a user that has fulfilled their goals using a health app, which should be seen as a natural part of the usage cycle. However, there may be the need to induce continued engagement to avoid relapse or maintain a given stage of evolution, which suggests that the app usage frequency pattern is as important as how the app is used by the user (52). Some apps might be designed for people to use them every day, whereas others might be designed for more emergent yet infrequent situations. O’Brien and Toms (56) in their conceptual framework of engagement with technology, drew attention to the cycles of engagement, disengagement, and reengagement that persists over time. The optimal “dose” of engagement is still unclear in the field of digital mental health interventions (53) and the time spent on a digital tool varies between different types of interventions, person, or context (57). Moreover, lack of engagement could be a good sign if it means that the users have achieved their recovery goals, or that the users are utilizing other resources that may be more helpful in each moment or that the app helped empower the users to seek traditional forms of therapy.
Many strategies are used to increase engagement, but the most common is the use of notifications that are a ubiquitous feature of mHealth apps, reminding the user to complete some action, informing about some event, or asking for information, which by itself increases the chances of data collection (58). However, the characteristics of message prompts have a huge impact on the effectiveness of this strategy, mainly the content, the design, or the frequency in which these prompts are shown to the user (59). So, care must be taken to not overuse notifications and to ensure that the notifications are relevant and personalized. Additionally, feedback is a strategy that is perceived as important and it is provided in various forms, including visual displays (e.g., graphs), report summaries, and messages.
Goal setting is one of the most used behavior change techniques in both face-to-face and digital health interventions as well as interactive content using different types of multimedia. Social support features motivate users by leveraging social influence (e.g., peer to peer communication). Offering rewards like points or badges in exchange for executing tasks or accepting challenges, is a common approach that incentivizes target behaviors. Mobile apps using gamification can be useful for well-being and mental health interventions and may enhance motivation and reduce attrition (60).
Some studies mention app personalization and customization as being important engagement factors; aesthetics also increases the friendliness of the design since a lack of aesthetics and text-heavy presentations may bore users. However, user experience does not predict sustained engagement with mental health apps (16). Likewise, usability is best understood as a potential barrier to engagement rather than something that increases engagement on its own. Usability is particularly important in digital health interventions for conditions in which users’ abilities may differ from the general population and the specific context where implementation is occurring, e.g., within a healthcare organization or directly to users.
On the other hand, there are factors that negatively influence user engagement, increasing the dropout of users such as the difficulty to use the technology, the value and usefulness perceived by the user or the adoption rate by population outside of studies (61). A person-centered approach or participatory design is needed to identify those factors, personal or app-specific, that may aid engagement at different stages. To produce an engaging intervention, it is critical to co-create and validate features with the people who will use them. Wang and Qui (62) presented in a systematic review the influencing factors of acceptance and use behavior of mobile health applications. They say that users’ behavior is influenced by individual demographic characteristics, app design, functionality, perceived ease of use and usefulness, security, cost, and personal motivations. Social attributes, source credibility, and legal issues also affect user behavior. Our results agree with these, insofar as in our scoping review some demographic factors were perceived as influencing adherence such as age, gender, education level and socioeconomic status. Younger people, women, more literate, and with higher socioeconomic status seem to show higher levels of involvement in this type of technological solutions.
Other factors also may play a role in influencing engagement: health status, data privacy and security, involvement in design process, incentives for participation, app usage fitting in the user routines, initial training, and constant technical support.
Regarding illness severity, some may argue that apps usage could be counterproductive or harmful on illness acute phases. SlowMo, one of the apps included in this review, specifically focused on addressing persistent paranoia which is common among patients with psychosis. This app is a digital intervention integrated with eight individual CBT sessions. The results showed lower levels of paranoia however these effects did not remain over time (43).
A faulty app that presents errors or bugs to the user can also have a negative impact in user engagement (63). In line with previous findings, the existence of human support is a critical aspect of digital interventions and reinforces the importance of the therapeutic alliance even in digital interventions (64, 65). Also, credibility can lead to more engaged users if the app incorporates expertise or clinician encouragement (21). A systematic review and thematic analysis found that personalization, reinforcement, and communication (communication can also be called social support and social network) are the design features mentioned the most often (66). Another systematic review (57) studied the barriers and facilitators of user engagement strategies with digital mental health interventions. The authors presented three categories—user-related factors, program-related factors, and factors related to the technology and implementation environment—and stated that by understanding these factors, targeted strategies can be used to overcome potential obstacles to the efficient use of digital interventions.
Regarding how to assess user engagement, there is high heterogeneity of criteria and measures used. This lack of consensus makes it difficult to compare results across studies and hinders understanding of what makes apps engaging for different users (18). It is also notable that interventions are generally described as being acceptable and feasible. Several methods to measure engagement were identified, including qualitative measures (semi-structured interviews, focus groups), self-report questionnaires, ecological momentary assessments, system usage data, sensor data, and clinical measures, which are consistent with those in previous literature (67). These comprise reports of the subjective user experience, elicited by qualitative methods or questionnaires (focusing on satisfaction questionnaires, interviews about usability, etc.), and objective measures of technology usage, user behavior, and users’ reactions to the intervention (focusing on usage frequency, response to prompts, trial retention). Self-reported measures are predominant in user engagement research and target subjective user engagement by way of users’ perception of technology mainly through post experience questionnaires. In addition, there are some validated instruments that can be used for these tools, specifically for research, for instance MARS—a framework that evaluates mobile health app quality along dimensions of engagement, functionality, aesthetics, and information quality (68). The engagement scale assesses how interactive and interesting the app is, the functionality scale assesses the app's functioning and ease of use, the aesthetics scale assesses overall visual appeal and stylistic consistency, and the information subscale assesses the quality of the content. Averaged together, the four subscales indicate the MARS total score, which measures overall app quality. None of the studies included in this scoping review used MARS. For real-world implementation, it is important to include in app features to automatically assess engagement and allow adjustments. It is also necessary to improve how stakeholders measure the impact of modifiable variables on engagement to understand the magnitude of effects (54).
O’Brien and colleagues (10) proposed an approach that could better meet users’ needs. In their perspective, design of apps would ideally be corroborative, outcome oriented, process based, and expert driven. In our perspective, app developers and mHealth service providers should use this study findings to include patients’ requirements in the app-development process.
Nevertheless, mobile health technology evaluation includes several dimensions, and user engagement is only one of them. The WHO mHealth Technical Evidence Review Group developed the mHealth evidence reporting and assessment (mERA) checklist. mERA aims to assist authors in reporting mHealth-research and the mHealth intervention (69) and this is very pertinent because there is a need for mHealth developers to follow guidelines for the buildup of mHealth apps, according to usefulness, quality, and safety standards. The American Psychiatric Association designed an evaluation hierarchal framework to guide informed decision making around the use of smartphone apps in clinical care (70), in which engagement is also included. This shared discussion among all stakeholders can also promote a more informed and sustained use of mobile applications in mental health.
Conclusion
Despite the potential benefits of mobile apps for people with schizophrenia, many interventions fail to engage their users effectively. Hence, our aim is to identify strategies and factors that may influence user engagement and facilitate prolonged use of apps for people with schizophrenia to better manage their condition. Nonetheless, there are some limitations in our study. The search was limited to articles published in English, as such, some articles may have been missed. In addition, the scoping review was not restricted to studies with high rigor, thus some of the report findings require replication since some studies lacked detail and this along with the variability in assessment tools made it challenging to summarize across studies. Because of this, a unified catalog to create a successful user engagement strategy was not achievable, which is something that is already clearer for other medical conditions. Also, there are apps which were not included in this scoping review because it did not have associated data.
Nonetheless, this scoping review provides some insights for increasing effective engagement in mobile apps for people with schizophrenia, analyzing not only user engagement strategies but also user-related factors, technology-related factors, and environment-related factors. Researchers, as well as practitioners, can use this knowledge to inform evaluations and development of new digital tools to support mental health services.
Future work should continue to explore innovative intervention strategies by which apps can support mental health in ways that appeal to people with schizophrenia, since a key component of the effectiveness of mental health apps is user engagement.
Author contributions
RSdA and AM contributed to the design and implementation of this scoping review, to the analysis of the results and to the writing of the manuscript. All authors contributed to the article and approved the submitted version.
Funding
This work was supported by Fundação para a Ciência e Tecnologia (FCT) through R&D Units funding (UIDB/05210/2020).
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1. Hoffman L, Wisniewski H, Hays R, Henson P, Vaidyam A, Hendel V, et al. Digital opportunities for outcomes in recovery services (DOORS): a pragmatic hands-on group approach toward increasing digital health and smartphone competencies, autonomy, relatedness, and alliance for those with serious mental illness. J Psychiatr Pract. (2020) 26(2):80–8. doi: 10.1097/PRA.0000000000000450
PubMed Abstract | CrossRef Full Text | Google Scholar
2. Firth J, Cotter J, Torous J, Bucci S, Firth JA, Yung AR. Mobile phone ownership and endorsement of “mHealth” among people with psychosis: a meta-analysis of cross-sectional studies. Schizophr Bull. (2016) 42(2):448–55. doi: 10.1093/schbul/sbv132
PubMed Abstract | CrossRef Full Text | Google Scholar
3. Young AS, Cohen AN, Niv N, Nowlin-Finch N, Oberman RS, Olmos-Ochoa TT, et al. Mobile phone and smartphone use by people with serious mental illness. Psychiatr Serv. (2020) 71(3):280–3. doi: 10.1176/appi.ps.201900203
PubMed Abstract | CrossRef Full Text | Google Scholar
4. Wong KTG, Liu D, Balzan R, King D, Galletly C. Smartphone and internet access and utilization by people with schizophrenia in South Australia: quantitative survey study. JMIR Ment Health. (2020) 7(1):e11551. doi: 10.2196/11551
PubMed Abstract | CrossRef Full Text | Google Scholar
5. Naslund J, Aschbrennerb KA. Technology use and interest in digital apps for mental health promotion and lifestyle intervention among young adults with serious mental illness. J Affect Disord Rep. (2021) 6:100227. doi: 10.1016/j.jadr.2021.100227
CrossRef Full Text | Google Scholar
6. Hoşgelen EI, Binnur Akdede B, Alptekin K. Prevalence use of technological devices and internet among patients diagnosed with schizophrenia and schizoaffective disorder. Schizophr Bull. (2020) 46(Suppl 1):S173–4. doi: 10.1093/schbul/sbaa030.413
CrossRef Full Text | Google Scholar
7. Lee K, Bejerano IL, Han M, Choi HS. Willingness to use smartphone apps for lifestyle management among patients with schizophrenia. Arch Psychiatr Nurs. (2019) 33(4):329–36. doi: 10.1016/j.apnu.2019.01.002
PubMed Abstract | CrossRef Full Text | Google Scholar
8. Jameel L, Valmaggia L, Barnes G, Cella M. Mhealth technology to assess, monitor and treat daily functioning difficulties in people with severe mental illness: a systematic review. J Psychiatr Res. (2022) 145:35–49. doi: 10.1016/j.jpsychires.2021.11.033
CrossRef Full Text | Google Scholar
9. Camacho E, Levin L, Torous J. Smartphone apps to support coordinated specialty care for prodromal and early course schizophrenia disorders: systematic review. J Med Internet Res. (2019) 21(11):e16393. doi: 10.2196/16393
PubMed Abstract | CrossRef Full Text | Google Scholar
10. O'Brien H, Morton E, Kampen A, Barnes S, Michalak E. Beyond clicks and downloads: a call for a more comprehensive approach to measuring mobile-health app engagement. BJPsych Open. (2020) 6(5):E86. doi: 10.1192/bjo.2020.72
CrossRef Full Text | Google Scholar
11. Torous J, Wisniewski H, Liu G, Keshavan M. Mental health mobile phone app usage, concerns, and benefits among psychiatric outpatients: comparative survey study. JMIR Ment Health. (2018) 5(4):e11715. doi: 10.2196/11715
PubMed Abstract | CrossRef Full Text | Google Scholar
12. Melbye S, Kessing LV, Bardram JE, Faurholt-Jepsen M. Smartphone-based self-monitoring, treatment, and automatically generated data in children, adolescents, and young adults with psychiatric disorders: systematic review. JMIR Ment Health. (2020) 7(10):e17453. doi: 10.2196/17453
PubMed Abstract | CrossRef Full Text | Google Scholar
13. Baumel A, Muench F, Edan S, Kane JM. Objective user engagement with mental health apps: systematic search and panel-based usage analysis. J Med Internet Res. (2019) 21(9):e14567. doi: 10.2196/14567
PubMed Abstract | CrossRef Full Text | Google Scholar
14. Ben-Zeev D, Scherer EA, Gottlieb JD, Rotondi AJ, Brunette MF, Achtyes ED, et al. Mhealth for schizophrenia: patient engagement with a mobile phone intervention following hospital discharge. JMIR Ment Health. (2016) 3(3):e34. doi: 10.2196/mental.6348
PubMed Abstract | CrossRef Full Text | Google Scholar
16. Kaveladze BT, Wasil AR, Bunyi JB, Ramirez V, Schueller SM. User experience, engagement, and popularity in mental health apps: secondary analysis of app analytics and expert app reviews. JMIR Hum Factors. (2022) 9(1):e30766. doi: 10.2196/30766
PubMed Abstract | CrossRef Full Text | Google Scholar
17. Cole-Lewis H, Ezeanochie N, Turgiss J. Understanding health behavior technology engagement: pathway to measuring digital behavior change interventions. JMIR Form Res. (2019) 3(4):e14052. doi: 10.2196/14052
PubMed Abstract | CrossRef Full Text | Google Scholar
18. Ng MM, Firth J, Minen M, Torous J. User engagement in mental health apps: a review of measurement, reporting, and validity. Psychiatr Serv. (2019) 70(7):538–44. doi: 10.1176/appi.ps.201800519
PubMed Abstract | CrossRef Full Text | Google Scholar
19. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med. (2017) 7(2):254–67. doi: 10.1007/s13142-016-0453-1
PubMed Abstract | CrossRef Full Text | Google Scholar
20. Torous J, Michalak EE, O'Brien HL. Digital health and engagement-looking behind the measures and methods. JAMA Network Open. (2020) 3(7):e2010918. doi: 10.1001/jamanetworkopen.2020.10918
PubMed Abstract | CrossRef Full Text | Google Scholar
21. Wu A, Scult MA, Barnes ED, Betancourt JA, Falk A, Gunning FM. Smartphone apps for depression and anxiety: a systematic review and meta-analysis of techniques to increase engagement. NPJ Digit Med. (2021) 4(20):1–9. doi: 10.1038/s41746-021-00386-8
PubMed Abstract | CrossRef Full Text | Google Scholar
22. Mustafa AS, Ali N, Dhillon JS, Alkawsi G, Baashar Y. User engagement and abandonment of mHealth: a cross-sectional survey. Healthcare. (2022) 10(2):221. doi: 10.3390/healthcare10020221
PubMed Abstract | CrossRef Full Text | Google Scholar
23. Arksey H, O'Malley M. Scoping studies: towards a methodological framework. Int J Soc Res Method. (2005) 8:19–32. doi: 10.1080/1364557032000119616
CrossRef Full Text | Google Scholar
24. Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. (2018) 169(7):467–73. doi: 10.7326/M18-0850
PubMed Abstract | CrossRef Full Text | Google Scholar
26. Achtyes ED, Ben-Zeev D, Luo Z, Mayle H, Burke B, Rotondi AJ, et al. Off-hours use of a smartphone intervention to extend support for individuals with schizophrenia spectrum disorders recently discharged from a psychiatric hospital. Schizophr Res. (2019) 206:200–8. doi: 10.1016/j.schres.2018.11.026
PubMed Abstract | CrossRef Full Text | Google Scholar
27. Ben-Zeev D, Brenner CJ, Begale M, Duffecy J, Mohr DC, Mueser KT. Feasibility, acceptability, and preliminary efficacy of a smartphone intervention for schizophrenia. Schizophr Bull. (2014) 40(6):1244–53. doi: 10.1093/schbul/sbu033
PubMed Abstract | CrossRef Full Text | Google Scholar
28. Ben-Zeev D, Brian RM, Jonathan G, Razzano L, Pashka N, Carpenter-Song E, et al. Mobile health (mHealth) versus clinic-based group intervention for people with serious mental illness: a randomized controlled trial. Psychiatr Serv. (2018) 69(9):978–85. doi: 10.1176/appi.ps.201800063
PubMed Abstract | CrossRef Full Text | Google Scholar
29. Ben-Zeev D, Brian RM, Aschbrenner KA, Jonathan G, Steingard S. Video-based mobile health interventions for people with schizophrenia: bringing the “pocket therapist” to life. Psychiatr Rehabil J. (2018) 41(1):39–45. doi: 10.1037/prj0000197
PubMed Abstract | CrossRef Full Text | Google Scholar
30. Hanssen E, Balvert S, Oorschot M, Borkelmans K, Os J, Delespaul P, et al. An ecological momentary intervention incorporating personalised feedback to improve symptoms and social functioning in schizophrenia spectrum disorders. Psychiatry Res. (2020) 284:112695. doi: 10.1016/j.psychres.2019.112695
PubMed Abstract | CrossRef Full Text | Google Scholar
31. Moitra E, Park HS, Gaudiano BA. Development and initial testing of an mHealth transitions of care intervention for adults with schizophrenia-Spectrum disorders immediately following a psychiatric hospitalization. Psychiatr Q. (2021) 92(1):259–72. doi: 10.1007/s11126-020-09792-9
PubMed Abstract | CrossRef Full Text | Google Scholar
32. Simões de Almeida R, Couto A, Sousa T, Marques A, Queirós C, Martins A. Development of weCOPE, a mobile app for illness self-management in schizophrenia. Arch Clin Psychiatry. (2019) 46(1):1–4. doi: 10.1590/0101-60830000000182
CrossRef Full Text | Google Scholar
33. Steare T, O'Hanlon P, Eskinazi M, Osborn D, Lloyd-Evans B, Jones R, et al. Smartphone-delivered self-management for first-episode psychosis: the ARIES feasibility randomised controlled trial. BMJ Open. (2020) 10(8):e034927. doi: 10.1136/bmjopen-2019-034927
PubMed Abstract | CrossRef Full Text | Google Scholar
34. Steare T, Giorgalli M, Free K, Harju-Seppanen J, Akther S, Eskinazi M, et al. A qualitative study of stakeholder views on the use of a digital app for supported self-management in early intervention services for psychosis. BMC Psychiatry. (2021) 21:31. doi: 10.1186/s12888-021-03317-9
PubMed Abstract | CrossRef Full Text | Google Scholar
35. Kidd SA, Feldcamp L, Adler A, Kaleis L, Wang W, Vichnevetski K, et al. Feasibility and outcomes of a multi-function mobile health approach for the schizophrenia spectrum: app4Independence (A4i). PLoS One. (2019) 14(7):e0219491. doi: 10.1371/journal.pone.0219491
PubMed Abstract | CrossRef Full Text | Google Scholar
36. Ben-Zeev D, Brian R, Wang R, Wang W, Campbell A, Aung M, et al. Crosscheck: integrating self-report, behavioral sensing, and smartphone use to identify digital indicators of psychotic relapse. Psychiatr Rehabil J. (2017) 40(3):266–75. doi: 10.1037/prj0000243
PubMed Abstract | CrossRef Full Text | Google Scholar
37. Bucci S, Barrowclough C, Ainsworth J, Machin M, Morris R, Berry K, et al. Actissist: proof-of-concept trial of a theory-driven digital intervention for psychosis. Schizophr Bull. (2018) 44(5):1070–80. doi: 10.1093/schbul/sby032
PubMed Abstract | CrossRef Full Text | Google Scholar
38. Lewis S, Ainsworth J, Sanders C, Stockton-Powdrell C, Machin M, Whelan P, et al. Smartphone-enhanced symptom management in psychosis: open, randomized controlled trial. J Med Internet Res. (2020) 22(8):e17019. doi: 10.2196/17019
PubMed Abstract | CrossRef Full Text | Google Scholar
39. Eisner E, Bucci S, Berry N, Emsley R, Barrowclough C, Drake RJ. Feasibility of using a smartphone app to assess early signs, basic symptoms and psychotic symptoms over six months: a preliminary report. Schizophr Res. (2019) 208:105–13. doi: 10.1016/j.schres.2019.04.003
PubMed Abstract | CrossRef Full Text | Google Scholar
40. Niendam TA, Tully LM, Iosif AM, Kumar D, Nye KE, Denton JC, et al. Enhancing early psychosis treatment using smartphone technology: a longitudinal feasibility and validity study. J Psychiatr Res. (2018) 96:239–46. doi: 10.1016/j.jpsychires.2017.10.017
PubMed Abstract | CrossRef Full Text | Google Scholar
41. Bonet L, Torous J, Arce D, Blanquer I, Sanjuan J. Remindcare app for early psychosis: pragmatic real world intervention and usability study. JMIR Mhealth Uhealth. (2020) 8(11):e22997. doi: 10.2196/22997
PubMed Abstract | CrossRef Full Text | Google Scholar
42. Terp M, Jørgensen R, Laursen BS, Mainz J, Bjørnes CD. A smartphone app to foster power in the everyday management of living with schizophrenia: qualitative analysis of young adults’ perspectives. JMIR Ment Health. (2018) 5(4):e10157. doi: 10.2196/10157
PubMed Abstract | CrossRef Full Text | Google Scholar
43. Garety P, Ward T, Emsley R, Greenwood K, Freeman D, Fowler D, et al. Effects of SlowMo, a blended digital therapy targeting reasoning, on paranoia among people with psychosis: a randomized clinical trial. JAMA Psychiatry. (2021) 78(7):714–25. doi: 10.1001/jamapsychiatry.2021.0326
PubMed Abstract | CrossRef Full Text | Google Scholar
44. Gire N, Caton N, McKeown M, Mohmed N, Duxbury J, Kelly J, et al. ‘Care co-ordinator in my pocket': a feasibility study of mobile assessment and therapy for psychosis (TechCare). BMJ Open. (2021) 11(11):e046755. doi: 10.1136/bmjopen-2020-046755
PubMed Abstract | CrossRef Full Text | Google Scholar
45. Moore E, Williams A, Bell I, Thomas N. Client experiences of blending a coping-focused therapy for auditory verbal hallucinations with smartphone-based ecological momentary assessment and intervention. Internet Interv. (2019) 19:100299. doi: 10.1016/j.invent.2019.100299
PubMed Abstract | CrossRef Full Text | Google Scholar
46. Schlosser DA, Campellone TR, Truong B, Etter K, Vergani S, Komaiko K, et al. Efficacy of PRIME, a mobile app intervention designed to improve motivation in young people with schizophrenia. Schizophr Bull. (2018) 44(5):1010–20. doi: 10.1093/schbul/sby078
PubMed Abstract | CrossRef Full Text | Google Scholar
47. Krzystanek M, Borkowski M, Skałacka K, Krysta K. A telemedicine platform to improve clinical parameters in paranoid schizophrenia patients: results of a one-year randomized study. Schizophr Res. (2019) 204:389–96. doi: 10.1016/j.schres.2018.08.016
PubMed Abstract | CrossRef Full Text | Google Scholar
48. Fulford D, Mote J, Gard DE, Mueser KT, Gill K, Leung L, et al. Development of the motivation and skills support (MASS) social goal attainment smartphone app for (and with) people with schizophrenia. J Behav Cogn Ther. (2020) 1:23–32. doi: 10.1016/j.jbct.2020.03.016
CrossRef Full Text | Google Scholar
49. Taylor KM, Bradley J, Cella M. A novel smartphone-based intervention targeting sleep difficulties in individuals experiencing psychosis: a feasibility and acceptability evaluation. Psychol Psychother. (2022) 3:717–37. doi: 10.1111/papt.12395
CrossRef Full Text | Google Scholar
50. Kreyenbuhl J, Record EJ, Himelhoch S, Charlotte M, Palmer-Bacon J, Dixon LB, et al. Development and feasibility testing of a smartphone intervention to improve adherence to antipsychotic medications. Clin Schizophr Relat Psychoses. (2019) 12(4):152–67. doi: 10.3371/CSRP.KRRE.070816
PubMed Abstract | CrossRef Full Text | Google Scholar
51. Lim MH, Gleeson JFM, Rodebaugh TL, Eres R, Long KM, Casey K, et al. A pilot digital intervention targeting loneliness in young people with psychosis. Soc Psychiatry Psychiatr Epidemiol. (2020) 55(7):877–89. doi: 10.1007/s00127-019-01681-2
PubMed Abstract | CrossRef Full Text | Google Scholar
52. O'Brien HL, Toms EG. What is user engagement? A conceptual framework for defining user engagement with technology. J Am Soc Inf Sci. (2008) 59(6):938–55. doi: 10.1002/asi.20801
CrossRef Full Text | Google Scholar
53. Saleem M, Kühne L, De Santis KK, Christianson L, Brand T, Busse H. Understanding engagement strategies in digital interventions for mental health promotion: scoping review. JMIR Ment Health. (2021) 8(12):e30000. doi: 10.2196/30000
PubMed Abstract | CrossRef Full Text | Google Scholar
54. Simblett S, Greer B, Matcham F, Curtis H, Polhemus A, Ferrão J, et al. Barriers to and facilitators of engagement with remote measurement technology for managing health: systematic review and content analysis of findings. J Med Internet Research. (2018) 20(7):e10480. doi: 10.2196/10480
CrossRef Full Text | Google Scholar
55. Palmier-Claus JE, Rogers A, Ainsworth J, Machin M, Barrowclough C, Laverty L, et al. Integrating mobile-phone based assessment for psychosis into people’s everyday lives and clinical care: a qualitative study. BMC Psychiatry. (2013) 13:34. doi: 10.1186/1471-244X-13-34
PubMed Abstract | CrossRef Full Text | Google Scholar
56. Graham AK, Kwasny MJ, Lattie EG, Greene CJ, Gupta NV, Reddy M, et al. Targeting subjective engagement in experimental therapeutics for digital mental health interventions. Internet Interv. (2021) 25:100403. doi: 10.1016/j.invent.2021.100403
PubMed Abstract | CrossRef Full Text | Google Scholar
57. Borghouts J, Eikey E, Mark G, De Leon C, Schueller SM, Schneider M, et al. Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. J Med Internet Res. (2021) 23(3):e24387. doi: 10.2196/24387
CrossRef Full Text | Google Scholar
58. Druce KL, Dixon WG, McBeth J. Maximizing engagement in mobile health studies: lessons learned and future directions. Rheum Dis Clin North Am. (2019) 45(2):159–72. doi: 10.1016/j.rdc.2019.01.004
PubMed Abstract | CrossRef Full Text | Google Scholar
59. Alkhaldi G, Hamilton FL, Lau R, Webster R, Michie S, Murray E. The effectiveness of prompts to promote engagement with digital interventions: a systematic review. J Med Internet Research. (2016) 18(1):e6. doi: 10.2196/jmir.4790
CrossRef Full Text | Google Scholar
60. Litvin S, Saunders R, Maier MA, Lüttke S. Gamification as an approach to improve resilience and reduce attrition in mobile mental health interventions: a randomized controlled trial. PLoS One. (2020) 15(9):e0237220. doi: 10.1371/journal.pone.0237220
PubMed Abstract | CrossRef Full Text | Google Scholar
61. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res. (2017) 19(11):e367. doi: 10.2196/jmir.8775
PubMed Abstract | CrossRef Full Text | Google Scholar
63. Kumar D, Tully LM, Iosif AM, Zakskorn LN, Nye KE, Zia A, et al. A mobile health platform for clinical monitoring in early psychosis: implementation in community-based outpatient early psychosis care. JMIR Ment Health. (2018) 5(1):e15. doi: 10.2196/mental.8551
PubMed Abstract | CrossRef Full Text | Google Scholar
64. Szinay D, Perski O, Jones A, Chadborn T, Brown J, Naughton F. Perceptions of factors influencing engagement with health and well-being apps in the United Kingdom: qualitative interview study. JMIR Mhealth Uhealth. (2021) 9(12):e29098. doi: 10.2196/29098
PubMed Abstract | CrossRef Full Text | Google Scholar
65. Tremain H, McEnery C, Fletcher K, Murray G. The therapeutic alliance in digital mental health interventions for serious mental illnesses: narrative review. JMIR Ment Health. (2020) 7(8):e17204. doi: 10.2196/17204
PubMed Abstract | CrossRef Full Text | Google Scholar
66. Wei Y, Zheng P, Deng H, Wang X, Li X, Fu H. Design features for improving mobile health intervention user engagement: systematic review and thematic analysis. J Med Internet Res. (2020) 22(12):e21687. doi: 10.2196/21687
PubMed Abstract | CrossRef Full Text | Google Scholar
67. Short CE, DeSmet A, Woods C, Williams SL, Maher C, Middelweerd A, et al. Measuring engagement in eHealth and mHealth behavior change interventions: viewpoint of methodologies. J Med Internet Res. (2018) 20(11):e292. doi: 10.2196/jmir.9397
PubMed Abstract | CrossRef Full Text | Google Scholar
68. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. (2015) 3(1):e27. doi: 10.2196/mhealth.3422
PubMed Abstract | CrossRef Full Text | Google Scholar
69. Agarwal S, LeFevre AE, Lee J, L'Engle K, Mehl G, Sinha C, et al. Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. Br Med J. (2016) 352:i1174. doi: 10.1136/bmj.i1174
CrossRef Full Text | Google Scholar
70. Torous JB, Chan SR, Gipson S, Kim JW, Nguyen TQ, Luo J, et al. A hierarchical framework for evaluation and informed decision making regarding smartphone apps for clinical care. Psychiatr Serv. (2018) 69(5):498–500. doi: 10.1176/appi.ps.201700423