Digital NCAD Mobile Application

Evaluating the Perspectives of Irish Student Nurses and Preceptors of an ICT System to Support Learning and Assessment in Clinical Practice

Major research project completed as part of an MSc in User Experience Design at IADT. First Class Honours received.

Forms & documentation comprising the NCAD

The National Competency Assessment Document (NCAD)

The NCAD is a paper based system for student nurses in Ireland that is used to plan and assess their learning on clinical practice placement.

The NCAD is filled out by the student nurse and the preceptor. The preceptor is a nurse who has undergone preceptorship training to support and assess the student nurse on the placement.

Understanding problems with the the current paper NCAD through discovery research

Phase 1

Phase 1 involved discovery research, whereby primary research was conducted with student nurses and preceptors. Qualitative research methods were employed, including semi-structured interviews. The aim of this phase of the research was to further specify the problems that student nurses and preceptors experience with the paper NCAD.

Competitor analysis was conducted on international digitised student nurse learning and assessment systems, e-learning mobile applications and relevant indirect competitors. The insights collected in Phase 1 helped to refine the research questions and informed the design of a digital prototype of the NCAD in Phase 2.

Ideating solutions with key stakeholders through a participatory design process

Phase 2

Phase 2 involved the design of a digital prototype of the NCAD using a Participatory Design Process with student nurses and preceptors. This was informed by the issues identified in the literature with the current NCAD, such as it being laborious and time consuming and containing complex language and terminology.

Co-design workshops were used to engage the key stakeholders in the design of the early stage prototypes and allowed for the identification of user needs and preferences. This informed the design and development process of the digital NCAD prototypes.

Evaluating the effectiveness of a digital solution through evaluative user research

Phase 3

Phase 3 involved the testing, evaluation and measurement process of the digital NCAD prototype with student nurses and preceptors. The insights gathered in phase 3, about the digital NCAD prototype, were compared to the insights gathered in phase 1, about the paper NCAD.

Phase 3 involved qualitative research methodologies with student nurses and preceptors. 10 moderated usability tests , post-test interviews and SUS evaluations on the digital NCAD prototypes.

Discovery Research Findings

One-to-one semi-structured interviews were conducted with the key stakeholders. Twelve interviews lasting one hour each were conducted, which comprised five student nurses, five preceptors and two subject matter experts.

The interviews explored the thoughts, feelings and behaviours of the key stakeholders relating to the paper NCAD. Thematic data analysis was performed and three themes and nine sub-themes emerged. (see below)

Theme 1: NCAD is Impractical

  • 1.1: Risk of Loss and Damage
  • 1.2: Missing Signatures in the NCAD
  • 1.3: Risk of forgery

    Theme 2: NCAD Lacks Value

    • 2.1: Repetative & Generic Domains of Competence
    • 2.2: Lacks Practical Guidance 
    • 2.3: NCAD not used appropriately
    • 2.4: NCAD viewed as tickbox exercise

      Theme 3: NCAD does not capture progress

      • 3.1: Preceptors can’t view student’s progress over time 
      • 3.2: Universities unable to monitor students remotely

      Opportunities with ICT

      What is Information Communication Technology (ICT)?

      ICT is a diverse set of technological tools and resources used to transmit, store, create, share or exchange information. Including: computers, mobiles, the internet, apps & websites. 

      Opportunities, as identified in the international literature, include the use of Information Communication Technology (ICT) for student nurse learning planning and assessment on clinical practice placement. The benefits of the use of ICT include enhanced learning and enhanced clinical competency assessment. 

      Enhanced Learning

      International studies have found that the use of e-learning mobile applications can lead to enhanced knowledge and skill test scores in student nurses.

      Enhanced Assessment

      Accessing information about the student’s previous placements helps preceptors to assess student competence, providing evidence of the student’s previous learning experiences and achievements.

      Research Problem & Questions

      International literature on the use of technological devices and applications to support clinical learning and assessment is well represented, however, these studies have focused mostly on the United Kingdom.

      To date, no such studies have taken place in Ireland. This research proposed to design and test a digital version of a sample of the NCAD with student nurses and preceptors in order to ascertain what benefits can be gained, compared with the paper based system that is currently in use.

      Efficiency

      Do Irish student nurses and preceptors perceive the use of an ICT system to be more efficient to support clinical learning and assessment, in comparison to the paper based NCAD system?

      Usability

      Do Irish student nurses and preceptors find the use of an ICT system to be more useable to support clinical learning and assessment, in comparison to the paper based NCAD system?

      Satisfaction

      Do Irish student nurses and preceptors find the use of an ICT system to be more satisfying to support clinical learning and assessment, in comparison to the paper based NCAD system?

      Methodology

      This study comprised two phases in which an A/B evaluation took place using qualitative methologies.

      (A = Paper NCAD, B = Digital NCAD Prototype)

      Phase 1

      Phase 1 involved discovery research, whereby primary research was conducted with student nurses and preceptors. Qualitative research methods were employed, including semi-structured interviews. The aim of this phase of the research was to further specify the problems that student nurses and preceptors experience with the paper NCAD. Competitor analysis was conducted on international digitised student nurse learning and assessment systems, e-learning mobile applications and relevant indirect competitors. The insights collected in Phase 1 helped to refine the research questions and informed the design of a digital prototype of the NCAD in Phase 2.

      Phase 2

      Phase 2 involved the design of a digital prototype of the NCAD using a Participatory Design Process with student nurses and preceptors. Participatory Design is a process by which end users are involved in the creation activities of a product or service [43]. This was informed by the issues identified in the literature with the current NCAD, such as it being laborious and time consuming and containing complex language and terminology [6, 7, 11]. Co-design workshops were used to engage the key stakeholders in the design of the early stage prototypes and allowed for the identification of user needs and preferences [25]. This informed the design and development process of the digital NCAD prototypes.

      Interviews

      One-to-one semi-structured interviews were conducted with the key stakeholders. Twelve interviews lasting one hour each were conducted, which comprised five student nurses, five preceptors and two subject matter experts.

      Thematic analysis was performed using Miro, and key findings were extracted. Themes and sub-themes were identified and are as follows: 

      Personas

      Following the primary research, the researcher created two primary personas, a student nurse and a preceptor. The personas capture the wants, needs and problems expressed by these user groups in the interviews. These insights were used to inform an improved digital NCAD prototype.

      Co-Design Workshops

      As part of a participatory design process, a series of co-design workshops were implemented with the key stakeholders using Miro. 

      The co-design workshops were conducted with the same participants from the interviews. They acted as follow-on discussions, whereby the issues identified in the interviews were discussed in more detail. The insights gathered from the interviews in Phase 1 were used to inform early-stage user flows and storyboards that were used to gather participant feedback and preferences for the digital NCAD prototypes.

      User Preferences

      In summary, participants voiced their preferences for the design of the Digital NCAD prototype as follows.

      Both students and preceptors remarked that they’d like:

      • To see their student/preceptor’s name(s) (respectively) and information about them (place of study/work, year of study/years qualified)
      • For the student’s completed Self Evaluation and Preliminary Interview to be accessible to the preceptor in advance of the placement
      • For the student to view common medications, skills and conditions that are common to the area of the placement.

       

      Preceptors would also like to be able to see the student’s previous NCADs and what placements they have completed. Uniquely, students wanted to see:

      • Information about the ward, such as start times and amenities e.g. if the ward has a fridge they can use, where to go on breaks etc.
      • Their roster/shifts,
      • All items within the NCAD are accessible from the dashboard,
      • Entries can be saved as a draft before final submission.

       

      Paper Prototyping

      Following Discovery research and co-design workshops, a series of paper prototypes were developed, focusing on the typical tasks a user undertakes using the paper NCAD, with improvements for the digital NCAD.

      Two views of the system were developed, the student’s view and the preceptor’s view. Content and functionalities differ between the two due to the different roles that they have.

      Student Nurse’s View

      Preceptor’s View

      High Fidelity Prototyping

      Student Nurse’s View

      1.0 Dashboard

       

      As informed by student preferences in the co-design workshops, upon login the student can access all NCAD items for the placement from the dashboard. The primary navigation in the app is a slider menu which is accessed from the hamburger menu icon.

      2.0 Self-Evaluation

       

      The student accesses the Self Evaluation exercise from the Dashboard. The student can save the exercise as a draft, this choice is related to the behaviors expressed by the students in the research, in that they complete this exercise over multiple sittings.

      3.0 Preliminary Interview

       

      The student can access the Preliminary Interview from the Dashboard. A list of suggested skills is provided to the student, tailored to their specific year of training and the clinical site. The student adds additional objectives. Once they have completed the Preliminary Interview, the preceptor can review it and add comments/suggestions.

      4.0 Middle Interview

       

      In the Middle Interview, the student is challenged to rate their progress towards achieving their previously selected learning outcomes on a five-point Likert scale. Instructions for how to use the rating scale are given via a key. The student can save as a draft or submit it to their preceptor for review.

      5.0 Final Interview

       

      In the Final Interview, the student can view their previous self-assessment of progress towards achieving their learning outcomes (from the middle interview). Once the student has submitted the final interview, the preceptor is notified and can complete their assessment of the student.

      Preceptor’s View

      1.0 Dashboard

       

      Upon logging in, the Preceptor is presented with their current student’s NCAD items, in the order in which it must be completed. The primary navigation takes the preceptor to all other sections in the app, such as their own Preceptorship History and Teaching Resources. These sections are informed by preferences stated by preceptors in the co-design workshops.

      2.0 Reviewing the Student’s Self-Evaluation

       

      In the Self Evaluation, the preceptor can read about the students relevant previous learning experiences and concerns about the upcoming placement. The desire to be able to access this information in advance of the placement was expressed by preceptors in the co-design workshops.

      3.0 Preliminary Interview

       

      In the Preliminary Interview, the preceptor can see what the student wants to learn on the upcoming placement and can suggest new learning objectives via the ‘add comment/suggestion’ function. Preceptors in the primary research said that they’d like to be able to suggest objectives as they have familiarity with the learning opportunities for students within their specific clinical area.

      4.0 Middle Interview

       

      In the Middle Interview, the Preceptor can view the student’s self-assessment of their progress thus far on the placement. The Preceptor can provide their evaluation of the student’s progress via the 1-5 Likert scale. The inclusion of the Likert scale is informed by participant preferences expressed in the co-design workshops. Instructions for how to use the Likert scale is included. The Final interview works in the same way, whereby the student completes their self-assessment, which is followed by the Preceptor’s evaluation of the student.

      Testing, Evaluation & Measurement

      Overview

       

      Following the design phase, a qualitative Testing, Evaluation and Measurement Process took place with the digital NCAD prototype. This took the place of an Experiment which might take place in a quantitative study. This process consisted of a series of usability tests and post-test interviews with five student nurses and five preceptors. Each group were given different views of the prototype dependent on role, the student’s view and the preceptor’s view.

       

      Set Up

       

      Moderated remote usability testing was conducted with Microsoft Teams as the participants lived in different parts of the country. Each session lasted between forty five minutes to one and a half hours and spanned over a two week period to accommodate the busy schedules of participants.

       

      Procedure

       

      The participants were asked to complete a series of tasks using the prototype that mimicked tasks within the paper NCAD, such as completing a Preliminary, Middle and Final Interview. Following the usability test, A System Usability Scale was completed by each participant to measure satisfaction with the prototype. Lastly, each participant took part in a post-test interview with the researcher.

      Analysis

      Thematic data analysis was performed in Miro via affinity diagramming. Student nurse and preceptor data were analysed separately and the positive and negative themes were colour coded to easily segregate the findings.

      Student Nurses

      Preceptors

      Results

      Following analysis of the Testing, Evaluation and Measurement data, the researcher evaluated the hypotheses against the data. The three hypotheses related to efficiency, usability and satisfaction. The next section will discuss the research findings in relation to the hypotheses. 

      H1 – Irish student nurses and preceptors perceive the use of an ICT system to be more efficient to support clinical learning and assessment, in comparison to the paper based NCAD system.

      H2 – Irish student nurses and preceptors find the use of an ICT system to be more useable to support clinical learning and assessment, in comparison to the paper based NCAD system.

      H3. Irish student nurses and preceptors perceive use of an ICT system more satisfying to support clinical learning and assessment, in comparison to the paper based NCAD system.

      H1 – Efficiency

      H1 – Irish student nurses and preceptors perceive the use of an ICT system to be more efficient to support clinical learning and assessment, in comparison to the paper based NCAD system.

      Estimated time on task for completing an interview in the paper NCAD was collected in the Discovery interviews in Phase 1 and again in the final stage Testing, Evaluation and Measurement Process in Phase 2 for the digital NCAD prototype.

      Both groups estimated that the length of time it would take to complete an interview using the digital NCAD is less than the time it takes for the paper NCAD.

       

      Student Nurse Estimated Time on Task Comparison

      Paper NCAD = average 28 minutes

      Digital NCAD = average 8 minutes

      Preceptor Estimated Time on Task Comparison

      Paper NCAD = average 27 minutes

      Digital NCAD = average 10.5 minutes

      Supporting Quotes

      H2 – Usability

      H2 – Irish student nurses and preceptors find the use of an ICT system to be more useable to support clinical learning and assessment, in comparison to the paper based NCAD system.

      Qualitative feedback on the usability of the paper NCAD was collected in the Discovery Interviews in Phase 1 and again in the final stage Testing, Evaluation and Measurement Process in Phase 2 for the digital NCAD prototype. A System Usability Scale (SUS) was used to capture quantitative feedback on the prototype with all 10 participants.

      Both groups reported higher levels of percieved usability with the digital NCAD and less with the paper NCAD. 

      Student Nurse SUS Score = 78.5/100

      Preceptor SUS Score = 83.5/100

      Interpreting SUS

      Bangor et al (2008) suggest a framework for the interpretation of SUS scores that is as follows, <50 = not acceptable, 50-70 is marginal and >70 is acceptable.

      In a publication by Sauro (2010) the author commented: “A [SUS] score of about 66 suggests that the application is below average in perceived usability, and a score above 66 is above average”.

      Therefore, scores of 78.5/100 and 83.5/100 are positive with regards to students’ and preceptors’ perceived usability of the digital prototype.

      Conclusion

      In conclusion, all of the three research hypotheses were retained and supported by the data collected in the research, H1, H2 and H3. Therefore, the results of this study support the following statements:

       

      1. Irish student nurses and preceptors perceive the use of an ICT system to be more efficient to support clinical learning and assessment, in comparison to the paper based NCAD system.

      2. Irish student nurses and preceptors find the use of an ICT system to be more useable to support clinical learning and assessment, in comparison to the paper based NCAD system.

      3. Irish student nurses and preceptors perceive use of an ICT system more satisfying to support clinical learning and assessment, in comparison to the paper based NCAD system.