The National Competency Assessment Document (NCAD) is a paper based system for student nurses in Ireland that is used to plan and assess their learning on clinical practice placement in Ireland. The purpose of this study was to explore the perspectives of Irish student nurses and Preceptors on the use of the mobile app prototype version of the NCAD and evaluate perceived efficiency and usability against the paper NCAD system.

This work was completed as part of my Major Research Thesis Project as part of my Master’s program in User Experience Design at IADT

Responsible for end-end research and design

October-May 2023

International literature on the use of technological devices and applications to support clinical student nurse learning and assessment is well represented, however, these studies have focused mostly on the United Kingdom.
To date, no such studies have taken place in Ireland. This research proposed to design and test a digital version of a sample of the NCAD with student nurses and preceptors in order to ascertain what benefits can be gained, compared with the paper based system that is currently in use.

As someone who was once a student nurse in Ireland and had first hand experience using several NCAD documents over the course of my 4.5 year degree programme, I saw an opportunity for digitisation of a traditionally paper based system. I wanted to explore if this would be possible and how the primary users (namely student nurses and their preceptors) would perceive it. So I chose to explore this as my thesis topic for nine months during my master’s degree.

This research study attempted to answer the following questions throughout the course of the research:

Do Irish student nurses and preceptors perceive the use of an ICT system to be more efficient to support clinical learning and assessment, in comparison to the paper based NCAD system?

 Do Irish student nurses and preceptors find the use of an ICT system to be more useable to support clinical learning and assessment, in comparison to the paper based NCAD system?

This study followed the Double Diamond Process Model (British Design Council, 2005) and was conducted in three phases and involved a qualitiative A/B evaluation (A = paper NCAD, B = digital NCAD prototypes). 

Phase 1 focused on discovery, using qualitative methods such as semi-structured interviews with student nurses and preceptors to identify issues with the paper-based NCAD. Competitor analysis of digital learning systems and apps further informed the research. Insights from this phase refined the research questions and guided the development of a digital NCAD prototype.

Phase 2 involved designing the digital prototype through Participatory Design with key stakeholders. Co-design workshops addressed issues found in the literature—such as complexity and inefficiency—and helped identify user needs. Inclusive and Universal Design Principles were applied, reflecting the diverse NCAD user base in Ireland.

Phase 3 involved the development of a high-fidelity prototype and evaluation with student nurses and preceptors using usability testing, post-test interviews, and the System Usability Scale (SUS), measuring efficiency and usability.

Twelve one-to-one semi-structured interviews were conducted with the key stakeholders, which comprised five student nurses, five preceptors and two subject matter experts, exploring attitudes, feelings and behaviours relating to the paper NCAD. Thematic data analysis was performed and three themes and nine sub-themes emerged. (see below)

1.1: Risk of Loss and Damage
1.2: Missing Signatures in the NCAD
1.3: Risk of forgery

2.1: Repetative & Generic Domains of Competence
2.2: Lack of Practical Guidance 
2.3: NCAD not used appropriately
2.4: NCAD viewed as tickbox exercise

3.1: Preceptors can’t view student’s progress over time 
3.2: Universities unable to monitor students remotely

Following the primary research, two personas were creates, a student nurse and a preceptor. The personas capture the wants, needs and problems expressed by these user groups in the interviews. These insights were used to inform an improved digital NCAD prototype.

As part of a participatory design process, a series of co-design workshops were implemented with the key stakeholders remotely using Miro.

The co-design workshops were conducted with the same participants from the interviews. They acted as follow-on discussions, whereby the issues identified in the interviews were discussed in more detail. The insights gathered from the interviews in Phase 1 were used to inform early-stage user flows and storyboards that were used to gather participant feedback and preferences for the digital NCAD prototypes.

Both students and preceptors remarked that they’d like:

1. To see their student/preceptor’s name(s) (respectively) and information about them (place of study/work, year of study/years qualified)
2. For the student’s completed Self Evaluation and Preliminary Interview to be accessible to the preceptor in advance of the placement
3. For the student to view common medications, skills and conditions that are common to the area of the placement, to help them to prepare in advance.

Uniquely, students mentioned they’d like to see:

1. Information about the ward, such as start times and amenities e.g. if the ward has a fridge they can use, where to go on breaks etc.
2. Their roster/shifts,
3. All items within the NCAD are accessible from the dashboard,
4. Entries can be saved as a draft before final submission.

Following Discovery research and co-design workshops, a series of paper prototypes were developed, focusing on the typical tasks a user undertakes using the paper NCAD, with improvements for the digital NCAD.
Two views of the system were developed, the student’s view and the preceptor’s view. Content and functionalities differ between the two due to the different roles and unique needs they have.

Following a round of guerilla usability testing with my target audience, the designs were taken to high fidelity to be tested as part of my final experiment.

Again, two views of the prototypes were created, the student’s view and the preceptor’s view.

As informed by student preferences in the co-design workshops;

Dashboard: Quick access to all required NCAD documentation for the current placement.

Progress: View completed NCADs from previous placements.

Learning Resources: Access tutorials and guides tailored to the placement area.
My Preceptors: View and message current and past preceptors.

Placement Schedule: Overview of yearly placement and university learning blocks.

Ward Info: Practical details like shift times and available amenities.

As informed by student preferences in the co-design workshops;

Self-Evaluation: Can be started in advance and saved as a draft, allowing completion over multiple sittings before submission.
Preliminary Interview: Provides suggested skills tailored to the student’s year and placement area, with the option to add custom objectives. Preceptors can review and comment after submission.

Middle Interview: Students rate their progress on a 5-point scale, offering more detailed self-assessment. Preceptors are notified once submitted.

Final Interview: Students reflect on preceptor feedback from the middle interview and complete their final self-assessment. Preceptors are then prompted to complete their final evaluation.

As informed by preceptor’s preferences in the co-design workshops;

Dashboard: Instant access to all required NCAD documentation for current student placements.

Preceptorship History: View current and past students, with messaging functionality.

Teaching Resources: Video guides on teaching relevant skills, tailored to student level and clinical area, helping clarify scope of practice.

Preceptorship Schedule: Yearly overview of student assignments, helping preceptors plan ahead and request breaks to avoid burnout.

Student Contacts: Easy access to key support contacts (e.g., CPC, university staff) for guidance during placements.

As informed by preceptor’s preferences in the co-design workshops;

Self-Evaluation Review: Preceptors can review the student’s prior learning experiences and concerns before placement begins.

Preliminary Interview Review: See the student’s learning goals and suggest additional objectives, based on clinical area opportunities.

Middle Interview Review: View the student’s progress via self-assessment and provide feedback using a 5-point Likert scale, with guidance included.

Final Interview: Similar to the middle interview; student self-assesses, followed by preceptor’s final evaluation.

After the design phase, a qualitative evaluation of the digital NCAD prototypes was conducted in place of a traditional quantitative experiment. This involved usability tests and post-test interviews with five student nurses and five preceptors, each interacting with role-specific views of the prototype.

Participants completed tasks mirroring those in the paper NCAD (e.g., Preliminary, Middle, and Final Interviews). Following testing, they completed the System Usability Scale (SUS) to assess usability and gave estimated time on task and then participated in post-test interviews.

The data was analysed to evaluate two hypotheses related to efficiency and usability.

H1 – Irish student nurses and preceptors perceive the use of an ICT system to be more efficient to support clinical learning and assessment, in comparison to the paper based NCAD system.

Estimated time on task for completing an interview in the paper NCAD was collected in the Discovery interviews in Phase 1 and again in the final stage Testing, Evaluation and Measurement Process in Phase 2 for the digital NCAD prototype.

“I think the rating system is actually much more efficient” P3

“I actually really liked the rating scale aspect, it’s a lot quicker.” P4

“Yeah, it would make it much quicker. It’s good. It would probably savev time, which is what we all want.” P3

“They (students) don’t have to write out everything. It’s a lot less time consuming.” P1

Both groups estimated that the length of time it would take to complete an interview using the digital NCAD is less than the time it takes for the paper NCAD.

H2 – Irish student nurses and preceptors find the use of an ICT system to be more useable to support clinical learning and assessment, in comparison to the paper based NCAD system.

Qualitative feedback on the usability of the paper NCAD was collected in the Discovery Interviews in Phase 1 and again in the final stage Testing, Evaluation and Measurement Process in Phase 2 for the digital NCAD prototype. A System Usability Scale (SUS) was used to capture quantitative feedback on the prototype with all 10 participants.

Both groups reported higher levels of percieved usability with the digital NCAD and less with the paper NCAD. 

Students and preceptors combined estimated time on task using the paper NCAD is 55 minutes. Students and preceptors combined estimated time on task using the digital NCAD is 18 minutes. As a % value, this amounts to an 62% increase in overall perceived efficiency.

Preceptors SUS score for the digital NCAD is 83.5 and student SUS score is 78.5. Bangor et al (2008) suggest that a SUS score above 66 is considered above average in terms of perceived usability.

I was awarded the prestigious Student of the Year Award in 2023 for my academic achievements in my Master’s program. I achieved a distinction in my final major research project and was awarded a first class honours degree.

The small sample size (10 participants) was a limitation of this study. Due to this, the researcher was unable to get any statistically significant findings from the quantitative method used in this study, the System Usability Scale. The research did however, uncover rich qualitative data rooted in the personal experiences of the participants, who were carefully selected as experts on this topic. Another limitation of this study is the use of a prototype. Prototypes are inherently limited in simulating how a person might interact with a functioning app in a real life situation, thus limiting the observations of the perceived experience. 
Due to the scope of this study, the researcher largely relied upon self-report from participants. A longitudinal, contextual inquiry study design may have yielded more valid and reliable results, whereby direct observations in the field could have been collected and analysed. 

One major constraint of this study was getting access to the right participants to participate in research and co-design activities over the course of the project. My participants had to be either studying or practicing in Ireland and were difficult to get access to due to the nature of their busy work schedules (e.g. shift work, weekend work, nights etc). In order to overcome this, I leaned on my own personal network of nurses and students and advertised for my research using the NMBI website and public forums like reddit and Facebook.

Another constraint is that I wanted to get representation of participants across the country to participate in the research, to do this I had to mostly conduct my research online. I would have loved to have held the co-design workshops in person to enhance creativity and engagement with the exercises but this was not possible.

In terms of success measurement, I was unable to capture a valid time on task for an interview using the actual paper NCAD due to difficulties with simulating this realistically. Instead, I had to rely on participants self-reported estimated time on task for comparison. For the same reasons, I was unable to capture a benchmark SUS score for the paper NCAD, which ultimately led to me having no baseline to compare the digital SUS scores to.

This qualitative study has generated questions beyond the scope of this research. Future research should aim to design, develop and test a functioning NCAD mobile application with the target user groups in a range of clinical settings. This study aims to inform a contextual longitudinal pilot trial of an application of this kind to evaluate its effectiveness, usability and satisfaction in practice. 
This research has generated insights into how the paper NCAD is perceived as suboptimal by its users, and the potential for greater effectiveness and usability with a digital NCAD application. Preceptors have indicated that they need more guidance on how to teach students and greater knowledge of scope of practice, a standardised method for assessing competency as a spectrum, a more efficient, less time consuming process. Students have indicated that they need a safe space to store placement documentation and clinical site specific guidance for skills they need to prepare them to become competent and safe practitioners. In the words of one student, “make it less of a tick box exercise and more about active participation in patient care.”