Diabetes is one of the most common major health conditions. It afflicts over 400 million people worldwide, and its prevalence is increasing (International Diabetes Federation, 2017). It is also costly both in terms of human suffering, with side effects such as kidney failure and blindness, and economically, with the International Diabetes Federation estimating that in 2017 world spending for diabetes, and related complications for adults, exceeded 850 billion USD. Effective treatment is challenging, depending not only on medications, but also continual patient engagement with factors such as emotional coping, exercise, diet, and problem solving; therefore, seeking means to support diabetes management is not constrained to clinical research, but a multi-disciplinary effort to support diverse aspects of care.
Despite the promise and excitement surrounding new technologies, there are significant challenges in designing systems that meet the multiple requirements posed by real-world Type 1 diabetes self-management. While we are optimistic about future technologies, it is essential to retain a critical viewpoint, for example questioning implicit design assumptions that require persistent manual logging and that automation of data collection will be sufficient without improving interaction design to better support user needs for situated decision-making and emotional sensitivity. In addition, current approaches often fail to support the cognitive processes used for diabetes self-management. And while the technical innovation could better meet such user needs, such systems could also introduce increased concerns about privacy, trust, and personal safety which must be carefully addressed.
In order to overcome such limitations, it is essential to design systems that solve actual user needs and augment user abilities, ideally forming a collaborative relationship which helps users to meet their personal goals. To accomplish this, designers must come to terms with how users interact with their data, their decision-making processes, their concerns about using such systems, and finally method of helping designers to create systems that meet those needs.
The following research aligns ideologically with the growing movement asserting that it is critical that technologies support the needs and rights of the users, rather than being exclusively means of achieving adherence to clinical metrics. It therefore follows that such tools should enhance independence and autonomy to the extent desired by the user. This calls for a balanced approach that recognizes that as well as helping to meet evidence based goals for glycemic management, users of diabetes technologies have a right to live their lives as they see fit, that diabetes management should not overly dominate lives, and that systems must be carefully considered as to their impact on user self-determination. Therefore, automation and other methods that help attain clinical targets and reduce the burden of diabetes management must be carefully balanced with the user maintaining conscious engagement and personal awareness when designing personal informatics tools.
Publications
Designing for Diabetes Decision Support Systems with Fluid Contextual Reasoning
Honorable Mention CHI ’18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
Data, Data Everywhere, and Still Too Hard to Link: Insights from User Interactions with Diabetes Apps
CHI ’18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
Unpublished work
DUETS: A Card-based Tool for Designing User-centered, Emotionally Sensitive, and Transparent Health Systems
ABSTRACT
Health apps that rely on manual logging and effortful reflection have often proven insufficient for meeting user needs. Networked pervasive health systems offer new opportunities to overcome inherent limitations; however, the complexities of these next generation systems pose challenges to system designers. This paper describes the design and evaluation of DUETS, a card-based system to support designers of digital health systems. We review relevant literature to derive criteria and questions for system assessment, followed by a description of the iterative design process for DUETS. Through focus group sessions with a health analytics startup and potential users, we assess both an early stage diabetes app prototype and evaluate the DUETS approach. The initial session resulted in the startup confronting significant concept flaws and pivoting their concept. We discuss the findings and wider implications for DUETS and suggest ways to apply the methodology to the design of diverse health management systems.
INTRODUCTION
The use of ubiquitous technology in the everyday management of health is growing rapidly. Advances in sensor capabilities combined with decreasing costs for technology, internet access, cloud computing, and mobile devices are allowing laypeople to use these technologies to track their health, compare their experiences to others, look up medical information online, and measure vital signs that were formerly accessible only in a clinical setting. These converging developments are changing the way people are informed about their health, and advancing methods of self-care for many chronic health conditions such as epilepsy [31], hypertension [32], asthma [22] and Type 1 (T1) diabetes – the use-case for this paper. Among major health conditions, diabetes is one of the most common. It afflicts over 400 million people worldwide and its prevalence is increasing [9]. T1 diabetes which affects 5-10% of those with diabetes, is an auto-immune condition requiring frequent injection of insulin to balance glycemic levels. Effective treatment is challenging, depending not only on medication, but also continual patient engagement with factors such as emotional coping, exercise, diet, and problem solving [5]. While smartphone apps offer multiple capabilities which could help support T1 diabetes management practices [14], there are substantial challenges to adoption [19] and studies have shown only modest clinical benefits [35]. In order to increase the adoption and efficacy of diabetes apps, it is essential to implement systems that function within the complex eco-systems of user’s lives [36]. Systems must reflect how users want to, and are capable of, interacting with their data, as well as supporting personal goals. Supporting future design in an efficient and systematic manner will require the development of effective criteria and methods for assessing the relevant complex eco-systems of devices and stakeholders, as well as practical methods of implementing emergent concerns and requirements. To build upon prior research in pursuit of these goals, we first review existing resources for the design and evaluation of user-centered computing systems for health, and then propose assessment criteria tailored to pervasive health systems. We then review the iterative design of the DUETS card-based system for health eco-system design and evaluation. In order to evaluate DUETS, we conducted focus group sessions with developers from a health analytics startup, and then potential patient users. In the final part of the paper we discuss the results of these focus groups, potential improvement in methods, and suggest means of modification for other chronic use-cases.
HEALTH and HCI
HCI has long researched the use of ubiquitous computing technologies to support healthier lifestyles. Intille [10] in his proposal for a UbiHealth workshop promoted sensors and mobile technologies to provide ‘just-in-time’ reminders to promote healthier behaviors; Fish’n’Steps [16] researched motivation through virtual fish on public and private monitors linked to pedometer step counts; ; UbiFit Garden [4] classified sensor data to automatically display activities within a virtual garden; Storni [30] explored the use of tagging to assist the user’s acquisition of additional contextual information, and Owen et al. [27] explored the addition of rich information such as photographs for reflection. However, for the most part, current diabetes apps still rely on manual data entry and effortful reflection as a primary means of supporting self-management. This is problematic in that such practices are difficult to maintain [1] and many users find such visualizations challenging to translate into actionable information [11]. Next generation approaches could reduce these barriers through increased use of networked sensors to collect data, combined with analytics to reduce cognitive effort [15] thereby increasing the efficacy of these apps. However, while personal data collection and visualization are readily available with current generations of wearables and apps, next-generation systems with more active decision support are still largely absent from the consumer space [26]. Consequently, there is still a lack of knowledge on how to develop such systems to meet user needs.
Designing next generations systems
As T1 diabetes management depends primarily on self-care [6], it is crucial that systems are oriented towards patient needs. User-Centered Design (UCD), emphasizing simplicity, ease of use, transparency of state, and clarity of purpose [25] provides a promising conceptual framework for diabetes system development. UCD asserts that users should be integrated into development, not just for their observations, but also as contributors of design ideas and solutions [7]. In this way, user participation can help realize products that more accurately respond to user needs, overcoming deficiencies of domain knowledge and misconceptions on the part of system developers [7]. Furthermore, since new technologies can impact vital rights such as privacy and autonomy [33], it is critical that such issues be addressed during the development process to avoid harm to the user, and damage to trust and reputation for the developer [34]. At the same time, UCD relies on iterative cycles of design, assessment, and improvement, which require some form of measurement to validate design choices [7]. Nielsen et al. [24] asserted that there are essentially four approaches to assessment, formal analysis techniques, automated procedures, empirical user experiments, and heuristic judgements. Nielsen et al. noted the practical limitations of the first two, and that empirical evaluations are intensive in time, labor, and expertise, thus advocating the use of heuristics. Persuasive health technologies pose particularly significant challenges for assessment, as they can require working prototypes to assess reactions, and extended deployment to enable realistic assessment, given that long-term adoption can be a critical factor in effectiveness [13]. In addition, due to the potentially hazardous nature of diabetes care, actual implementations of diabetes decision support systems face high regulatory and ethical barriers. Therefore, lightweight methods such as heuristics that can assist developers to identify concerns within early stage systems could be of considerable benefits to iterative design processes in pervasive health.
Criteria for system assessment
In this section we suggest five criteria for examination of multi-component health support systems. The first, Adoption & Utility considers specific hardware and software functionalities and their fitness for purpose [23]. The next three are derived from the ABC model of Attitudes (Affect, Behavior, and Cognition). Finally, Consent & Control, reflects the sensitivity and potential risks to the user posed by personal health data [2]. Each category will now be discussed:
Adoption & Utility: Systems features should be designed to support and empower the user to achieve personally relevant goals by means compatible with their lifestyle [36].
Emotional Impacts: Personal health data can, in certain situations, bring about feelings of stigma or vulnerability [3]. In such situations, inappropriate UI design elements can reinforce counterproductive emotions, reducing engagement, or causing system rejection [11]. Therefore, it is critical that designers minimize negative emotional impacts.
Behaviors & Actions: Effective diabetes management requires diverse behaviors to optimize outcomes [5]. Systems should therefore support such behaviors while minimizing over-reliance on technology that can inhibit beneficial and self-sustaining habit formation [29].
Cognitive demands: Diabetes care requires awareness and decision-making [28]. Systems should be designed to support contextually appropriate cognitive processes [12].
Consent & Control: External behavior and biometric monitoring poses privacy and security risks to the user [34], and therefore should be responsive to user desires for transparency and control over their personal data [13]. Taken together these criteria form a proposed framework for identifying key user concerns for systems to support the self-management of chronic health conditions. These categories and their criteria, appropriately expanded, form the basis for the Concerns cards in the DUETS system.
Assessment frameworks
As previously alluded to, Nielsen et al. [24] promoted heuristics for efficient, practical concern identification. Kientz et al. [13] built upon Nielsen’s work, developing a modified set of heuristics aimed specifically at persuasive health technologies. However, these heuristics are not well suited to assessing some key aspects of complex health management eco-systems, for three reasons identified by Wright [31] and colleagues [32]. Firstly, when context is critical, questions can yield more flexible and context-sensitive assessment than heuristics [31]. Secondly, while privacy is mentioned in the Kientz et al. heuristics, questions can help to further investigate the embedded privacy and ethical values which are essential when designing and developing new technologies [34]. Thirdly, information flows are a critical factor in assessing such risks. This third issue suggests that developing methods to help stakeholders to understand these flows could play an important role in supporting improved design processes. However, integrating these requirements poses certain challenge in implementation such as: the diversity of user data which can be captured by ubiquitous sensor equipped devices, the hard-to-predict insights that might be implicit in such flows, and the possibilities of such data being shared or sold to external third parties. As a further challenge, when systems and data processing steps evolve through the development and implementation process, there is a need for practical methods of iterative reassessment by relevant stakeholders.
Combining the Concerns criteria with the question-based approach
Combining the Concerns criteria with the question-based approach outlined above helps to suggest a revised basis for the iterative assessment and design of persuasive health technologies. Eight aspects summarize this revised approach: integrating users throughout the project; embracing iterative and incremental process; multiple stakeholders to gather diverse viewpoints; use of questions to help identify key concerns; tailoring questions to the use-case as required; privacy as an important concern of technology; embracing an ethical perspective that places priority on patient needs; and finally integrating information flows and helping users understand system function as an essential aspect of the discussion process. The combination of the Concerns criteria with the question-based approach as outlined above provides a refined basis for system evaluation and design. The next section discusses ways of applying this new approach.
Practical methods of application: Support for card-based approaches
In the context of supporting the iterative design and assessment of multi-component pervasive diabetes decision support systems, we now consider how to apply the above approach in practical situations. Given the need to deal with multiple stakeholders, foster engagement, illuminate complex information flows and balance diverse viewpoints, one candidate method of application is a card-based system. For example, in the context of IoT systems, Mora et al. [21] suggested that card-based tools are useful for fostering engagement, communication, inspiration, flexibility, collaboration, diversity of opinion, and externalization of thoughts. In the context of collaborative design, Halskov & Dalsgard [8] favored a card -based approach on the grounds that tangible objects assist design processes by focusing and supporting constructive interactions. Luger et al. [18] promoted the use of cards as a design instrument to help students and designers understand often inaccessible legal regulations in regard to data protection and privacy. Finally, Lucero et al. [17] developed playful experiences cards to facilitate brainstorming and scenario building. Generally, card-based systems appear well suited to supporting collaboration, ideation, and integration. However, to the best of our knowledge, cards have not elsewhere been applied to modeling and reflection specifically tailored for health system architectures.
DUETS: the design process and components
This section outlines the iterative design process used to create the card system. Key features and changes leading to the version of DUETS used in trials (described later) are outlined, followed by a review of the DUETS components.
Version1 (Fig.1): Taking a ‘minimal viable product’ approach, the first prototypes featured hand written architecture cards. Initial trial sessions resulted in feedback that system structure was too hard to understand, and cards were too visually similar. Version2 (Fig.2): The second version used hexagons to represent devices and stakeholders. These nodes were then connected with arrows to show information flows. Version3 (Fig.3): Color coding was added in order to visually differentiate node types. Hexagons replaced with octagons to allow more information about connections. People and companies differentiated visually. Information flow arrow cards given blank spaces to add attributes. Tokens (small cut out pieces of paper) with different words used to prompt relevant details, and space created on nodes for the placement of relevant Concerns cards. Version4 (Fig. 4): Tokens adopted for connectors to prompt reflection on information flows. Cards and a scenario tested with game designers, who suggested using circular nodes in order to allow more flexible system architecture construction, and white tokens to increase readability. Version5 (Fig. 5): Colored outlines used on tokens, to indicate category. Trialed with a startup developer, assessing their current product. Drew attention to specific security concerns with data flows between certain components, and other design flaws. However, complexity of the many API’s and other services used within their system took up too much space for a standard sized table, suggesting the need for a more compact and space efficient design. Version6 (Fig. 6): Size of the nodes and connector cards reduced. Colors of the nodes re-designed, people cards received more inclusive skin-tone gradient. Squares tokens to increase font size in proportion to total space needed and decrease trimming time. System used to assess an IoT system with lead designer from an IoT startup, who suggested representative graphics would be more intuitive and that the basic architecture construction was inappropriate. Participant became disengaged during the time required to locate and place pre-printed attribute tokens, suggesting the need for better usability. Version7 (Fig. 7): In response to the criticisms in the previous session, the next iteration was influenced by the design choices of [21]. In addition, checklists with tick boxes replaced tokens to increase efficiency. This stage of the prototype was used for focus groups and will be discussed in more detail in the following section. All versions, including newest version are available at http://bit.ly/2xD8ljV.
Components of DUETS
The following section describes the categories and content of the implemented DUETS cards: Concerns Cards which raise questions for reflection and discussion; Architecture Cards used to describe the components of a multi-component system; and Attributes Cards, which list information that might impact the relationships between stakeholders.
Concerns Cards (Fig. 9)
Identifying user concerns can be an important aspect of developing user-centered systems [24]. The Concerns cards follow the criteria and application framework discussed in previous sections. The questions on each card were designed to provoke reflection and discussion leading to the identification of domain relevant concerns and consideration in the following eight areas respectively. Adoption & Utility: product features that could impact value and workload for the user and their willingness to integrate products into daily self-management practices. Emotional Impacts: systems or UIs might be emotionally supportive or alternately produce stigma, discouragement, counter-productive emotions, or be insensitive within specific contexts. Behaviors & Actions: self-care behaviors are essential throughout a range of daily activities. Cognitive demands: consideration of whether systems place contextually appropriate cognitive requirements on users. Consent & Control: data usage, transparency of process, and capability of users to maintain control and distribution over personal information.
Architecture Cards (Fig. 10-12)
The Architecture cards support visualizing multi-component systems, helping users to consider which devices, services, stakeholders, and companies make up a proposed system and the relationships between them. Movable cards can also enable convenient modifications and provoke suggestions for alternative configurations. All cards contain space for note taking or the marking down of relevant attributes. The four categories are as follows. Devices: hardware which make up systems including consumer products such as smartphones and smartwatches, medical devices, and data storage. Services: The eco-system of user supporting services such as health coaching, peer-support, clinical care, education, analytics, and monitoring that might be integrated into the system design. Stakeholders: individuals who might be exchanging information or services within a described system – this includes patients, family, peers, co-workers and anyone to whom the patient may have a personal relationship. Companies: commercial enterprises, these stakeholders have been separated from Stakeholders to draw attention to differences in personal and legal obligations.
Attributes Cards (Fig. 13)
The Attribute cards list specific relevant details which might help participants to understand or consider the relationships between nodes in the architecture, as well as arrows to assist in considering direction of information flows, asymmetric relationships, or implied dependencies. The cards include checkboxes to prompt consideration and notation of relevant details. The six categories are as follows. Data: collected information that is moved during transactions. Goals: motives or hoped for outcomes of different stakeholders. Actions: activities that are occurring as part of normal system function. Traits: health conditions or other qualities which might need special considerations. Emotions: feelings that could occur with or during system use. Responsibilities: duties or demands placed on stakeholders.
Assessing the DUETS system
This section discusses methods and procedures used within three focus group sessions conducted to assess the utility of the DUETS system. The initial session assessed an early stage app concept with a diabetes AI startup, the second session assessed the same app with a patient group, and the final session returned to the startup to assess results. The startup had already engaged in over 2 years of sensor data classification and analytics research and were looking to implement their algorithms into a consumer facing product. To this end they had produced a product description and three screen shots of a mocked-up smartphone app, offering the following benefits: Alerting the user to adverse events or patterns detected in data (Fig. 14); Alert the user to irregular activities (activity, sleep, location), and offer insights (Fig. 15).; Predict future behaviors (activity, sleep) based on habits and offer suggestion (Fig. 16).
Procedures & methods
At the beginning of the sessions, each participant received a multi-page form with a series of questions relating to concerns and suggestions to improve the system. The procedure was as follows: Prior concerns: after a briefing, participants noted any existing concerns with the app concept based on their prior knowledge. Reflection with heuristics: Participants were asked to read through the Kientz et al. [13] heuristics and independently note any new concerns. Reflection with Concerns cards: Participants used the DUETS Concerns cards and again noted any newly identified concerns with the product. Visualizing system with DUETS Architecture cards: The components of the product were tangibly visualized using the Architecture and Attributes cards, and participants then sorted again through the Concerns cards, placing them at relevant point on the architecture and noting any new concerns. Feedback: Participants reflected on the process and gave feedback. This multi-step procedure was designed to assess the added value of each stage by the new concerns generated. The heuristics [13] were chosen as a control since they were developed specifically for assessing persuasive health technologies. Following the stated protocol of [13], each participant was given the heuristics on a single printed sheet. At each stage throughout the workshop, participants were asked to write down new observations, together with the heuristic or concern card that had prompted this observation and note any ideas or product suggestions. As per the Nielsen protocol, participant responses were combined, and duplicates removed. For the developer sessions, there were no incentives offered other than the value to their development process, while each member of the user group was offered a 30 Euro Amazon gift card for participation. Ethics approval was granted by the university ethics committee.
First and third focus group: Developer team
The session took approximately 2 hours to complete. Participants were all members of the product development team: P1 a software engineer, P2 a backend engineer, and P3 an upper level manager of the company with a computer science background. All three were male with a mean age of 35 years (SD 6).
Second focus group: People with T1 diabetes
The second session took approximately 2 hours. Participants were all people with T1 diabetes (n=6), 5 female, 1 male, mean age 32.8 years (SD 14.1). All participants were familiar and engaged with diabetes self-management technology.
Results of focus groups
This section reviews the observations of the developer team as they were asked to reflect upon and note any concerns related to their app concept.
Session 1: Development team assesses the app concept
In this first section, the developer team were asked to note any existing concerns they had with their app concept.
Pre-existing concerns
Adoption & Utility : P1 noted “Information must be relevant”,“Real-time information must most likely need to be processed on the devices”, and “Medical recommendations are dangerous”. P2 noted that since current continuous glucose monitor manufacturers do not offer a real-time API, “… We can’t be as real time as we would like”, that there is a “lack of insulin data”, it being hard to “identify relevant pieces of information for different users”, and that “some cards display interesting information that might be not relevant”. P3 noted that “Information we can provide right now to app user is not relevant enough and does not solve a pain point”, and that due to the lack of real-time BG and insulin data “…a lot of the more interesting scenarios/notifications cannot be implemented right now” There were also concerns as to optimizing as “different diabetics may have wildly different needs and we might be optimizing for one single group” And that “Users do not want yet another app and (to) use multiple apps in parallel.” Emotional Impacts:The developer group noted some general concerns with the affective aspects of their app, with P1 noting that “Information might be annoying”, and that “Historical data seems overwhelming.” And P2 observed that “The apps can be annoying if it notifies too much.” Behaviors & Actions:There were limited concerns at this stage with P1 noting “Information must be actionable.” Consent & Control: P1 wrote that “Users have privacy concerns.”
Summary: In this first stage the developer team noted 12 concerns, demonstrating that they were aware of challenges to the viability of their concept app. They were especially concerned with the limitation placed by currently available technologies, and whether they would be able to deliver useful information, especially given diverse needs of different users. However, at this stage these concerns were often non-specific, and lacked clear paths forward for improvement.
Reflection with heuristics
In the second stage of this session the developer team were given a single page printout of the Kientz et al. heuristics and asked to read it over and write down any additional concerns.
Adoption & Utility: As with the first stage, the majority of noted concerns were specifically about the features offered by the app. P1 observed that “… we have only briefly considered positive feedback/gamification.” P2 noted “Currently our solution is totally automatic, so no user input is allowed”, and that therefore “Not using the phone would mean we don’t have any data.” P2 also became more aware of certain general missing functionalities, such as “User status not provided” and “No customizability.” There was also increased awareness of the company’s current limitations, as “Our knowledge about user interaction design is limited”, and “Our interface design capabilities are limited.” And the observation that “The technology is hard to validate.” P3 was concerned that “Our activity recognition is not 100% accurate, so it may provide wrong notification/information”, and that “There may be delays due to technology limitations and we will not provide the information at the right time + place.” Emotional impacts: P3 noted that “The user may not want to be bothered about their condition at all.” Cognition: P1 wrote “Parts of our technology might be hard to explain”, P2 was concerned about a lack of “…user education”, and P3 wrote “We stay on surface with information that truly promote learning and thereby does not engage the user.” Consent & Control:P2 expressed concerns that users might have unjustified concerns which could create resistance to adoption, “even though our App is completely anonymous, it can be perceived as a threat to privacy since it detects and logs all user actions and identifies patterns based on them.”
Summary: The heuristics helped identify 14 new concerns, drawing attention to some specific missing features such as “rewards”, “status” and “customizability”, and some aspects additional limitations of the notifications which could need further refinement. Also, while many of the initial concerns were domain specific, the heuristics added more general health technology concerns, especially identifying gaps in the team’s skillset. User education was noted as an area needing additional work, and threats to privacy were noted, though not considered an actual risk.
First focus group: Reflection with Concerns cards
In this stage the development team was given the DUETS Concerns cards and asked to note any additional emerging concerns that had not previously been noted.
Adoption & Utility: P1 observed that the app as designed had limited ability to incorporate user feedback to train the algorithm and questioned “How do we incorporate manual feedback for automatically recognized activities/habits?”, and questioned, “How bad is manual entry really?” P1 also noted that the “The product is entirely based on user data, it has a very cold start.” P2 added, “Product is meant to be used on a daily basis. Benefits can surpass required effort, but currently there’s no way of making the user aware of benefits.” And P2 wrote “Ideally it would be accepted by doctors. We are interviewing several diabeticians to check what can be relevant to them, but we need to put more effort on this matter.” And while the lack of customization had been noted in earlier stages, P2 noted “There is no user customization, and no justification.” P3 also more closely considered the lack of user customization, “We may need extensive customization options.” Emotional impacts:The Concerns cards helped the developers become more aware of the affective aspects of their product, and the need for better product research with users. P2 wrote, “Some cards could be stressing since they are warning about likely future adverse events.” P3 also became more aware of the limitations of their development process, as “We have not done any user testing yet how users react to notifications/information in real-life. We only interview people.” Behaviors & Actions: P1 observed “With great notifications come great responsibility (people might rely on us).” and that “Education might be undervalued by us” Cognition: P1 reflected “What are real problems for diabetics?” P3 added, “We provide little guidance on the effect of possible user actions after getting information from our system.” And that “The product will not work on its own, i.e. the user still needs another app see his/her historical data.” Consent & Control: P1 noted “Improvements to the medical outcome/diabetes management are even harder to validate.” And that “The technology only works if people actually use phone/watch.” And that “We should focus more on the patients.” And P3 wrote, “People may feel uncomfortable seeing tracked and a third-party app having all their behavior and medical data.”
Summary: The use of the Concerns cards helps identify potential product flaws across all categories, with 20 new concerns identified. While many aspects of these concerns had been previously noted, the concern cards appeared to produce more detailed observations and deeper consideration of how their product might impact the user. The need for greater emphasis on user research and involvement was also noted. In terms of features, the developers noted several detailed interaction challenges, such integrating manual feedback, and improving the onboarding process until the algorithms became trained. In terms of emotional impact, the developers noted the need for greater user-testing, especially on interface color choices, and stress related to predictions. Concerns directly related to user behavior were also added, noting the importance of user-education, and potential adverse effects of system use. The impact of the technology on user autonomy was also noted, which is a crucial ethical concern. In terms of user data, whereas the developers had only previously noted that users might have privacy concerns, there was more specific reflection on how such user discomfort might prevent the app from functioning. While many of these issues such as privacy of data were integrated into the Kientz et al. heuristics, the cards appeared to add more detail and broader consideration of how these concerns might interact with the user.
Visualizing system with DUETS Architecture cards
In this stage the researcher assisted the developer team in visualizing their proposed system with the Architecture cards (see Fig. 8). The Concern cards were then divided among the team, and they were then instructed to sort through the cards, placing them onto places on the architecture that seemed most relevant to specific concerns. Adoption & Utility: While there were 6 cards placed at various locations on the architecture, all these concerns had been addressed in previous stages and are therefore not discussed further. Emotional Impacts:The developer team noted 2 new concerns, placing the card related to stigma and user interfaces on the smartwatch card. While the team had previously noted the potential negative emotional effects of undesired predictions, at this stage they added to the patient card the concern card related to the negative effects of undesired actual information. While a subtle difference, this card has additional design implications: one relates to predictions alone, as opposed to encompassing the user’s own retrospective data. Cognition: The developers placed the card related to triggering the user’s existing knowledge on the smartphone and noted the importance of their service delivering visualizations that would minimize misinterpretation of collected data. Consent and Control:While in stage 1 and 2 the developers noted privacy concerns, the more specific Concern cards were placed on various components of the architecture. These included the card related to the use of personal data by 3rd parties; trust; and transparency related to stakeholders within the network. P1 reflected that “Privacy might be a bigger concern than I usually think.”
Summary: There were 10 new concerns noted and the use of the visualization appeared to help developers consider the ways in which users might have legitimate concerns about 3rd party data use. This drew attention to the need for increased transparency to promote user trust.
Developer Team Feedback:
The development team wrote the following feedback for the DUETS system. P2 stated, “I think this has been a really useful process in order to identify the system’s concerns. Having to think about the problems and let me find my actual concerns and put into perspective the ones I was focused on.” There was also critical feedback, P1 noted “Marking which data exactly flows from A to B seems useless for our case.” P3 suggested that noting which data was aggregated and/or anonymized should be better indicated. There was also the suggestion that participants be encouraged to create Concerns cards during the reflection process, and that the process should include more time for group discussion.
Summary of developer session
Each stage of the session added additional concerns, with the observations becoming more detailed and specific to the use-case. The Kientz et al. heuristics were successful in drawing attention to many concerns, especially in Adoption & Utility category. The Concerns cards delivered more detailed user-centered concerns, such as the need to engage stakeholders more directly in the development process, as opposed to the heuristics which tended to be interpreted by the developers as pointing towards the need for more staff. While the Architecture cards did assist with the identification of some concerns, the process was somewhat unengaging, and benefits were not clear cut. While there were some important concerns identified, especially with privacy vulnerabilities, the session suggested that the DUETS system was still in need of further improvement. This limitation could have come from the developer team already being familiar with system architecture. Now that we have reviewed the observations of the development team, the next section will review a follow-up session on the startup’s app concept with a focus group of people with diabetes (PWD).
Session 2: People with Diabetes evaluate app concept
As the startup decided after their initial session to undertake a major pivot on their product, this PWD section will be restricted to a summary of findings. However, parts of these sessions will feature, where relevant, in the subsequent discussion section.
Prior concerns: At this stage, the PWD user group identified 21 concerns. Of these, 14 had been identified by the developers in the prior-knowledge stage; 3 in the heuristics’ session, and 1 using the Concerns cards. Thus, this preliminary session chiefly duplicated concerns identified by the developer group, though at times in richer detail.
Heuristics: 12 new concerns were located, though once again all but 2 had been identified by the developers at some stage. Of these, the developers identified 6 with the use of the heuristics, and 2 with the use of the full DUETS system. The users found 2 new concerns: 3rd parties becoming annoyed by increased interaction; and risks of accidental sharing of data on social media – which was not an offered feature. User identified concerns continued to be more detailed and descriptive than developer concerns.
Concerns cards: At this stage, the users identified 36 new concerns, although 19 had been previously identified as concerns by the developer group. Many of the novel concerns were in the form of suggestions for new features, such as integration with clinical software. There were some newly stated concerns around the need to minimize workload for the user – which the developers might have missed, given the manual data entry already eliminated by the app. There were also concerns expressed about business models and developer motives, and the ability to operate system remotely without sharing data.
DUETS Architecture cards and concerns cards: In this last stage, in a departure from the originally planned protocol, the participants engaged in a 45-minute free flowing conversation prompted by questions read out loud by the moderator from the Concerns cards. This user discussion session produced 35 new concerns, with 23 of them not having been mentioned by the developer group. Of these, 11 were suggestions for new features. This session appeared especially useful for eliciting detailed observations and thoughts about new functionalities or interaction approaches. In general, the users appeared able to draw upon their experiences to give more detailed examples than the developers. For instance, while the developers questioned “How do we incorporate manual feedback for automatically recognized activities/ habits …?”, a user suggests “Are you cycling to work today” where is the ‘yes’ ‘no’ button. Just want to answer fast and put the mobile phone back in my pocket.” Or while a developer noted “The user may not want to be bothered about their condition at”, a user explained “…therapy has to get better and less thinking… otherwise (we) won’t see the reason to use it.” Or while a developer might be concerned that “Parts of our technology might be hard to explain”, a user provided insights towards desired answers “…how does it actually analyze these things? Like, what’s the algorithm? Who designed it? Why? With what goal in mind?”
Session 3: Developers review sessions
Upon the start of the session, the developer team stated that in the intervening 7 weeks since the initial session, their company had undergone a large pivot. P2 noted, “We changed the concept to focus on diabetes Type 2 and focus on habit formation. This happened as we felt the previous concept was not compelling enough and did not address a true need.” And P1 added “…Type 1 diabetes …we don’t know enough about it.” P3 listed that they had concerns that “Availability of data was not very good, Unclear what problems we were solving, T1 people have very heterogenous needs.” P3 noted that their first session “… helped catalyze our decision to shift our product concept.” Upon reading over the user responses, P3 observed that their initial session had successfully identified the majority of high-level issues, while the user session had offered deeper insights into features. P3 added, “I think getting this kind of user feedback can be very beneficial, but our old concept was in a very early stage. Maybe too early stage for this kind of feedback. I don’t think it was really clear what we wanted to achieve with our concept.” One of the developers noted that they thought the DUETS process would offer more benefits with frequent brief sessions.
Summary: The developer group was positive about the first session for helping them to realize the flaws and shortcoming of their app concept. They concluded that due to technical limitations beyond their control, the abundance of T1 requirements, and their insufficient domain knowledge, they should undergo a significant product pivot.
DISCUSSION
The following section discusses current strengths and weaknesses of the DUETS system as used in the focus groups.
DUETS
The Concerns cards appeared beneficial for generating concerns not previously located by the heuristics and seemed especially useful for eliciting and evidencing suggestions for new features, generally leading to more specific and detailed information. This could have been due in part simply to the use of questions rather than statements, though further research would be needed to distinguish between possible contributing factors. Of the total concerns identified in the developer and user focus groups, use of DUETS generated 26/52 in the former and 67/100 in the latter, though the extended group discussion in the user group may have been a contributing factor. Many of the concerns located with the use of DUETS were implicitly referenced in the Kientz et al. heuristics. For example, the concern related to “incorporating manual feedback” might be seen as implicit in heuristic 6, that states that the technology should allow manual editing or updating of data. And concerns that the app could interfere with autonomy might be inferred from heuristic 10, which refers to the need to educate users. Explanations for missing these concerns at the heuristics stage might be that participants lacked the knowledge to fully exploit the heuristics, or that the density of the heuristics inhibited their effective use. If the latter were true, breaking the heuristics down into shorter sentences on cards might help to alleviate this problem. This suggests that further research is needed to clarify the extent to which the Kientz et al. heuristics are well-suited to non-UX professionals. The DUETS tool was generally well-received by the developers, and they credited the process with helping them to confront critical gaps between their concept and a desirable product. This realization could benefit the developers, since the competition posed by the great number of diabetes apps already on the market heightens the value of ‘failing fast’ before significant investment. While the developer focus group only identified half as many concerns as the user group, the developers identified the majority of key concerns. Of the 53 unique concerns noted by the developer team, only 11 were not on the user list. Of these, many related to general user-centered practices (need for more user testing, increased attention user needs, involving medical professionals in development processes, integrating user education, challenges in explaining technology), others were technical (the need for on-device processing for real-time information, technology was phone dependent, means of using user input to train algorithm), and some interaction design challenges (User privacy concerns due to
misunderstanding product, Stress related to potential adverse future events, Triggering user’s existing knowledge). However, the majority of these concern are variations of concerns noted in the user-focus group and therefore it appears that the developer focus group was successful in locating user concerns.
While the user group did identify 48 concerns more than the developers, the majority of these were potential additional features, such as integrating retrospective data, or displaying uncertainty of predictions. Some of these user concerns were general, and either of a kind readily detectable in user-testing (ambiguity of message phrasing in prototypes), or marketing suggestions (target more specific niche groups, focus on watch interfaces.) The users did note some new emotional concerns (others might be annoyed by increased phone usage, or that machines simulating humans can be annoying). While the developers had been aware of privacy and security concerns, the users emphasized the need for transparency to promote trust: in business models, motives, and data control. Taken together, the primary added value of the user focus group was suggesting new features, together with specific and highly detailed interaction preferences, and emphasis of the need for transparency to build trust.
Given that the focus group for developers identified most of users’ concerns, it might be useful to engage user focus groups after initial concerns have been located and solved, and when developers are looking for more in-depth insights, or need to test their assumptions. The DUETS process helped the developers to understand design challenges inherent in the development of effective diabetes decision support systems. The initial use of DUETS by the developer group highlighted the need for in-depth understanding of the health domain, and for clear goals in terms of what actual problems are being addressed. These issues were clarified to such an extent that it became apparent to the developer team that they had neither the knowledge nor technical capacity to meet the actual needs effectively. Another design challenge identified by DUETS bound together three elements for the developers: the suspicion that products of this kind might face with users; the consequent need to actively build trust with users; and the need for clear statements of purpose and of means for achieving stated goals.
The use of the cards appears to assist developers in confronting their limitations and asking uncomfortable questions about the added value of their product. The sessions with users demonstrated effective support for generating ideas for new features, and in-depth analysis of concerns. Within the limitations of this preliminary evaluation (as itemized below), the DUETS system appears to provide a usable framework for this purpose.
Limitations
This study is an exploratory case study. No claim is made regarding a rigorous comparison between methods. The small sample size limits the generalizability of conclusions. The sessions did not randomize procedure order, which could influence the quality, quantity and ratios of identified concerns. The user group was an engaged and technology aware group, and their insights might not be broadly representative. The researcher brought prior knowledge which could have influenced outcomes; the utility and usability of the DUETS system without such support might be different. DUETS was developed for a diabetes use-case and therefore might require additional or different cardsfor other health conditions. Some apparent benefits of DUETS might be due simply to use of a question-based rather than a statement-based format. The added benefits of the Architecture cards as opposed to using the Concerns cards alone requires further investigation given the small sample size and variables. The proper assessment of the value of the DUETS system for helping to support more user-centered, emotionally sensitive, and transparent systems requires further development both for the cards and their method of integration into design processes. In general, the system needs further refinement to lower cognitive effort. Further testing and iteration are called for throughout.
Future Work
DUETS is an expandable modular system. The use of cards could be customized, for example by the inclusion of additional cards to reflect identified needs. Concerns identified by developers could be added to the Concerns deck either for in-house discussion after iteration, or to elicit feedback from other stakeholders. Likewise, Architecture cards could be easily added to, reflecting new devices or services. While DUETS supported concerns identification, further development is needed. The Kientz et al. heuristics elicited many concerns, especially with the developer team, and could be directly incorporated into the Concerns cards. The Behaviors & Actions category elicited the fewest Concerns and might benefit from additional cards or clarification. Despite apparent benefits, specifically for helping to draw attention to data privacy and transparency concerns, there was some inconclusiveness on the effectiveness of the Architecture cards, which might be addressed by further studies. This could be done with a between-group study, with one group only using Concerns cards compared to the other using Concerns cards an
Architecture cards. Given the bio-ethical implications of decision support systems [20], the framework proposed by Beauchamp & Childress [2] might be an alternative to the current criteria, with augmentation for privacy and security [33] and technical features. System usability might be improved by reducing the number of Concerns cards.
CONCLUSIONS
Through three sessions, the first and third with the developer team of a funded startup, and the second with a motivated user group, the DUETS tool was assessed for its ability to assist in the location of concerns which could impact the effectiveness of multi-component user-centered health systems. In the initial developer session, DUETS contributed to the team identifying, in under 2 hours, critical failings in their app concept, technical limitations, and domain knowledge which, by their own account, led to the decision to make a substantial pivot. Supporting such early strategic decision-making can be valuable, given the high costs associated with developing functional medical systems. The session with potential users validated the developer’s concerns, added in-depth experiences that supported the reasoning behind these concerns, and identified many suggestions for potential features. Within the limitations previously outlined, these sessions evidence how the DUETS tool can be used as a method both for developers to critically examine their assumptions and concerns for early stage products, and for potential users to offer critical and constructive critiques.
DUETS appeared successful in drawing attention to the high-level concerns related to functionalities, emotional impacts, and cognitive demands, as well as drawing attention to user privacy and security concerns, in particular emphasizing the need for product transparency. The DUETS process drew attention to aspects of requirements for user-centered design, such as greater integration with stakeholders and their needs, and clear identification of true user problems. Given that the DUETS tool helped the developers to locate many of the high-level concerns held by the user group, it may have a useful role in helping developers to locate user concerns during early stage development. Following subsequent prototype iteration, DUETS may provide utility in the design process to elicit and marshal in-depth critical stakeholder feedback. In summary, DUETS offers a modular card-based method, readily expandable and tailorable, that can be used to visualize and assess multi-component health systems that involve a complex ecology of devices, services, and stakeholders. By facilitating an overview of the information flows between components, stakeholders, and services, as well as considering resultant implications, this approach could support developers and other stakeholders in assessing relationships, identifying concerns, and suggesting new features for next generation health support systems.
REFERENCES
1. Eirik Arsand, Ragnhild Varmedal, and Gunnar Hartvigsen. 2007. Usability of a mobile self-help tool for people with diabetes: the easy health diary. In Automation Science and Engineering, 2007. CASE 2007. IEEE International Conference on, 863–868. Retrieved April 4, 2016 from http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4341807
2. Tom L. Beauchamp and James F. Childress. 2001. Principles of biomedical ethics. Oxford University Press, USA.
3. Betty P. I. Chang, Thomas L. Webb, and Yael Benn. 2017. Why Do People Act Like the Proverbial Ostrich? Investigating the Reasons That People Provide for Not Monitoring Their Goal Progress. Frontiers in Psychology 8. https://doi.org/10.3389/fpsyg.2017.00152
4. Sunny Consolvo, David W. McDonald, Tammy Toscos, Mike Y. Chen, Jon Froehlich, Beverly Harrison, Predrag Klasnja, Anthony LaMarca, Louis LeGrand, Ryan Libby, and others. 2008. Activity sensing in the wild: a field trial of ubifit garden. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1797–1806. Retrieved January 29, 2015 from http://dl.acm.org/citation.cfm?id=1357335
5. American Association of Diabetes Educators. 2009. AADE guidelines for the practice of diabetes self-management education and training (DSME/T). The Diabetes Educator 35, 3_suppl: 85S-107S.
6. Martha M. Funnell and Robert M. Anderson. 2004. Empowerment and self-management of diabetes. Clinical diabetes 22, 3: 123–127. Retrieved August 14, 2016 from http://clinical.diabetesjournals.org/content/22/3/123?cited-by=yes&legid=diaclin;22/3/123&patientinform-links=yes&legid=diaclin;22/3/123
7. Jan Gulliksen, Bengt Göransson, Inger Boivie, Stefan Blomkvist, Jenny Persson, and Åsa Cajander. 2003. Key principles for user-centred systems design. Behaviour & Information Technology 22, 6: 397–409. https://doi.org/10.1080/01449290310001624329
8. Kim Halskov and Peter Dalsgard. 2006. Inspiration card workshops. In Proceedings of the 6th conference on Designing Interactive systems, 2–11.
9. International Diabetes Federation. 2017. IDF diabetes atlas. International Diabetes Federation, Brussels, Belgium. Retrieved from http://www.diabetesatlas.org
10. Stephen S. Intille. 2004. Ubiquitous computing technology for just-in-time motivation of behavior change. Stud Health Technol Inform 107, Pt 2: 1434–1437. Retrieved January 26, 2015 from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.5.7011&rep=rep1&type=pdf
11. Dmitri S. Katz, Blaine A. Price, Simon Holland, and Nicholas Sheep Dalton. 2018. Data, Data Everywhere, and Still Too Hard to Link: Insights from User Interactions with Diabetes Apps. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 503.
12. Dmitri S. Katz, Blaine A. Price, Simon Holland, and Nicholas Sheep Dalton. 2018. Designing for Diabetes Decision Support Systems with Fluid Contextual Reasoning. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 625.
13. Julie A. Kientz, Eun Kyoung Choe, Brennen Birch, Robert Maharaj, Amanda Fonville, Chelsey Glasson, and Jen Mundt. 2010. Heuristic evaluation of persuasive health technologies. In Proceedings of the 1st ACM International Health Informatics Symposium, 555–564. Retrieved January 23, 2015 from http://dl.acm.org/citation.cfm?id=1883084
14. David C. Klonoff. 2013. The Current Status of mHealth for Diabetes: Will It Be the Next Big Thing? Journal of Diabetes Science and Technology 7, 3: 749–758. Retrieved September 10, 2014 from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3869144/
15. David C. Klonoff and David Kerr. 2018. Overcoming Barriers to Adoption of Digital Health Tools for Diabetes. Journal of Diabetes Science and Technology 12, 1: 3–6. https://doi.org/10.1177/1932296817732459
16. James J. Lin, Lena Mamykina, Silvia Lindtner, Gregory Delajoux, and Henry B. Strub. 2006. Fish’n’Steps: Encouraging Physical Activity with an Interactive Computer Game.
17. Andrés Lucero and Juha Arrasvuori. 2010. PLEX Cards: a source of inspiration when designing for playfulness. 28–37. https://doi.org/10.1145/1823818.1823821
18. Ewa Luger, Lachlan Urquhart, Tom Rodden, and Michael Golembewski. 2015. Playing the Legal Card: Using Ideation Cards to Raise Data Protection Issues within the Design Process. 457–466. https://doi.org/10.1145/2702123.2702142
19. Ananthidewi Maniam and Jaspaljeet Singh Dhillon. 2015. Barriers to the Effective Use of Diabetes Self-Management Applications. In Proceedings of the 3rd National Graduate Conference, 315–320.
20. Rob Meredith and David Arnott. 2003. On ethics and decision support systems development. In On Ethics and Decision Support Systems Development. PACIS 2003 Proceedings. Paper 106. Retrieved November 6, 2016 from https://works.bepress.com/david_arnott/1/download/
21. Simone Mora, Francesco Gianni, and Monica Divitini. 2017. Tiles: A Card-based Ideation Toolkit for the Internet of Things. 587–598. https://doi.org/10.1145/3064663.3064699
22. myAirCoach. 2016. myAirCoach. myAirCoach. Retrieved January 11, 2016 from http://www.myaircoach.eu/myaircoach/
23. Jakob Nielsen. 2012. Usability 101: Introduction to Usability. Nielsen Norman Group. Retrieved November 11, 2017 from https://www.nngroup.com/articles/usability-101-introduction-to-usability/
24. Jakob Nielsen and Rolf Molich. 1990. Heuristic evaluation of user interfaces. 249–256. https://doi.org/10.1145/97243.97281
25. Don Norman. 2016. The design of everyday things. Verlag Franz Vahlen GmbH.
26. Fredrik Ohlin and Carl Magnus Olsson. 2015. Intelligent Computing in Personal Informatics: Key Design Considerations. In Proceedings of the 20th International Conference on Intelligent User Interfaces, 263–274. Retrieved April 14, 2015 from http://dl.acm.org/citation.cfm?id=2701378
27. Tom Owen, Jennifer Pearson, Harold Thimbleby, and George Buchanan. 2015. ConCap: Designing to Empower Individual Reflection on Chronic Conditions using Mobile Apps. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services, 105–114.
28. Árún K Sigurðardóttir. 2005. Self-care in diabetes: model of factors affecting self-care. Journal of Clinical Nursing 14, 3: 301–314. https://doi.org/10.1111/j.1365-2702.2004.01043.x
29. Katarzyna Stawarz, Anna L. Cox, and Ann Blandford. 2015. Beyond self-tracking and reminders: designing smartphone apps that support habit formation. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 2653–2662.
30. Cristiano Storni. 2014. Design challenges for ubiquitous and personal computing in chronic disease care and patient empowerment: a case study rethinking diabetes self-monitoring. Personal and Ubiquitous Computing 18, 5: 1277–1290. https://doi.org/10.1007/s00779-013-0707-6
31. Nicola Swanborough. 2016. Husband programmes smartwatch to detect wife’s seizures| Epilepsy Society. Retrieved January 11, 2016 from http://www.epilepsysociety.org.uk/news/husband-programmes-smartwatch-detect-wifes-seizures-05-01-2016#.VpO40fEb5K0
32. Tactio Health. 2016. Home. Tactio Health Group | Science Based mHealth Solutions, Remote Patient Monitoring. Retrieved January 11, 2016 from http://www.tactiohealth.com/
33. David Wright. 2011. A framework for the ethical impact assessment of information technology. Ethics and Information Technology 13, 3: 199–226. https://doi.org/10.1007/s10676-010-9242-6
34. David Wright and Michael Friedewald. 2013. Integrating privacy and ethical impact assessments. Science & Public Policy (SPP) 40, 6: 755–766. Retrieved December 16, 2017 from http://libezproxy.open.ac.uk/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=a9h&AN=92875759&site=ehost-live&scope=site
35. Yuan Wu, Xun Yao, Giacomo Vespasiani, Antonio Nicolucci, Yajie Dong, Joey Kwong, Ling Li, Xin Sun, Haoming Tian, and Sheyu Li. 2017. Mobile App-Based Interventions to Support Diabetes Self-Management: A Systematic Review of Randomized Controlled Trials to Identify Functions Associated with Glycemic Efficacy. JMIR mHealth and uHealth 5, 3: e35. https://doi.org/10.2196/mhealth.6522
36. A Review of User-Centered Design for Diabetes-Related Consumer Health Informatics Technologies. Retrieved December 26, 2014 from http://dst.sagepub.com/content/7/4/1039.full.pdf