Using diary studies to gather feedback on the ‘Register with CQC’ service
Rose Edmands and Kelly Garlick, The Care Quality Commission.
During our Private Beta phase of the project ‘Register with CQC’, we have been looking into different ways to get feedback from our users. We want to make sure we speak to our users and gain insight from those who are going through the new service; understanding what is working well but also what isn’t, so that we can focus on making those improvements.
After reaching out to users who have consented to be contacted for research, we offer users different ways in which they can provide their feedback, based on their preference and availability. Feedback calls seems to be the most popular approach for users to provide their feedback and during Private Beta, we have completed 49 calls with users so far. This call can happen at various stages of registering; preparing supporting documents, after submitting the online application, after validation checks and/or when users are given a registration decision and certificate. This approach has provided us with rich feedback, in retrospect, to a given task or experience. This has its advantages, as it allows users to reflect on their experiences and consider what improvements may have helped them, thinking about the overall picture. However, it doesn’t allow us to capture feedback ‘in the moment’ or at other points of their journey. Therefore, another option that we offered users to provide feedback was to take part in a short diary study.
A diary study enables users to jot down their behaviours or experiences whilst they are completing the task in hand. It can mimic the same effects of being a fly-on-the-wall but doesn’t require the need for us to be there in person at the time (ideal during a pandemic!). With the CQC registration application being an important and often lengthy process for an organisation or individual, it is also not usually something that can be completed in one go. Therefore, the use of diary studies seemed ideal. We wanted to understand the motive of why users would log into their online application and what they’d like to achieve each time, how long they’d spend on the application on that day and any thoughts on what worked well for them or not so well.
For our diary study, we decided to offer users a word document template for them to fill out each time they’d visit their online application. As users would be on a desktop/laptop/mobile to research or complete the application, we thought it would be easier for users to complete the diary study this way, rather than jotting any experiences down on a physical notepad. We also asked users to send the diary studies as and when they completed one, so we could keep a track of where they were in the application and see how this evolved over time. Once they had submitted their registration application, users were given the option to take part in a feedback follow on call.
So far, we have eight users doing diary studies, five users have completed these and sent them back to us. And from these five users, we have 11 entries in total. As mentioned earlier, after users have completed their diary study and submitted the application (which could range from a few days to a few months), some individuals then stepped forward for a feedback call to talk through their experiences. The feedback received in the diary study has been very beneficial at helping frame the call and has allowed the user to go into more detail and expand on what they had written. Diary studies have also highlighted some factors that CQC are not responsible for, such as hardware corruption and losing information by not saving this, outside of the application itself. This has led to users experiencing delays when completing and submitting their application, however this is something we would not be aware of without feedback from diary study entries. Likewise, during current times when we can’t be physically with the users, we feel it has been the next best thing to gain this insight. From the 11 entries, we have also been able to draw themes on how long users generally spend on certain tasks, what has caused some issues (e.g. document upload on the application) and how users have tried to resolve these problems.
However, one drawback to a diary study is the retention rate and users forgetting to send back any diary study entries they have completed. Sometimes, we’ve had to prompt or follow up with users a few weeks to a month later to understand how they are getting on with their application. As ever, some users didn’t have the time to complete the diary study, so they ended up taking part in a feedback call only. Similarly, in our template we did not advise the amount of feedback or have a word count, so some entries are quite bare, making it harder to analyse and draw conclusions from. A couple of things to try in the future would be to communicate the expectation upfront, for example checking in on users every few weeks to see if they have any entries. Also, we could provide users with an example diary study entry to demonstrate the information we’d like to capture and therefore potentially get richer data from our users.
Another method of participation we have offered users for providing feedback during Private Beta is to take part in ‘live observation’. This is where we recommend users to share their screen with us whilst they complete the registration application in their own time. Again, as we currently can’t conduct contextual research in person, this would act as an alternative to observe users in their own time. Unfortunately, we haven’t had any volunteers for this approach, but something we could refine or suggest in the future would be to ask users if they would be happy to record themselves (e.g. using the Teams recording function) whilst they complete the application, over time. This could be part of the diary study and would act as further evidence to the notes they provide alongside this. It would allow us to understand what approach users take to completing the registration application and allow us to view a series of clips over a period of time. A combination of this, a diary study and a feedback call would enable us to get an even richer picture of someone’s journey whilst they complete the registration application with CQC.
Overall, the use of diary studies as part of our feedback on ‘Register with CQC’ have complemented our other approaches to gain user feedback during the Private Beta phase. They have brought attention to other issues we wouldn’t have been aware of and it’s allowed users to contextually note down their experiences at the time, rather than retrospectively during a call. There is always room for improvement to refine methods, which we’re looking to do in the coming months. One of the main things we’ve learnt is that diary studies don’t have to be formal or time-consuming and instead, they offer another way that users can provide their valuable feedback.
