Eat, Sleep, Research, Repeat? Conducting Cognitive Interviews In A Pandemic

by Alan Donnelly

The outbreak of COVID-19 has forced face-to-face research and evaluation activities across the institution and the sector to be postponed or carried out using different methods. A project at Sheffield Hallam aimed to conduct cognitive interviews with students as part of the development of a questionnaire to evaluate teaching and learning.  This blog post will complement existing recommendations on ‘Doing fieldwork in a pandemic’ (e.g. Lupton, 2020) by:

  1. considering whether technology can be utilised as a viable alternative to in-person participation for conducting cognitive interviews, such as by using phones and web-based video conferencing.
  2. outlining the approach of web probing, which takes place without an interviewer, as an emerging self-administered method of evaluating survey questions.
  3. exploring the factors that determine whether carrying out research activity is essential right now.

What is cognitive interviewing?

Cognitive interviewing refers to a set of techniques used to assess the ways in which individuals process and respond to survey questions (Willis, 2004), specifically with the aim of examining: how questions are interpreted and responded to; the consistency of these responses across all participants; the causes of any difficulties; potential solutions to resolve these issues. Two main techniques are used in cognitive interviews (Beatty & Willis, 2007):

  • ‘think-aloud’, which requires participants to describe their thought processes while responding to a question (for example, “tell me what you are thinking about?”)
  • verbal probing, where the interviewer takes more of an active role by asking participants follow-up questions about specific terms or responses (for example, “can you tell me what this word means to you?”).

‘Right Here, Right Now’?

The first consideration is to determine whether carrying out any research activity right now is essential:

    • There is a strong likelihood that members of the target population are ‘displaced, working from home, out of work, looking for jobs, juggling family responsibilities…unwell or taking care of people who are ill’ (Ravitch, 2020).
    • Conducting research activity has potential ethical implications for participants’ wellbeing, privacy and confidentiality, while the responses that they can provide might be influenced, such as by a lack of focus.
    • Consider whether there are any issues of equity in relation to participant access and representation, for example, whether the perspectives of any groups will be excluded by not being able to participate (Ravitch, 2020).
    • Researchers are instructed to undertake a risk assessment to identify the risks for the target population (Austen, 2020). If a decision is made to proceed, ethical approval will be required to conduct the research using the new method that have been agreed.

‘Weapon of Choice’: What are the delivery methods?

The preferred delivery method of conducting cognitive interviews is in face-to-face settings (Willis, 2004) because it allows observation of non-verbal cues, such as body language, and enables a social interaction to take place between the participant and interviewer without any interruptions.  The use of technology presents researchers with new possibilities and challenges for running cognitive interviews:

  • Technology can aid access to certain populations, such as those with a physical disability who might not be able to travel to a venue for face-to-face engagement, but act as a hindrance for other groups, for example, individuals with visual or cognitive impairment or people who do not have access to the internet.
  • There is the opportunity to test surveys with more varied and diverse samples which can help identify additional errors (Blair & Conrad, 2011), for example, technology can simplify the task of selecting individuals from specific demographic backgrounds.

A limited number of studies have carried out cognitive interviews which do not require in-person engagement (see Figure 1), such as by phone (e.g. Noel, 2013) and web-based video conferencing (e.g. Dean, Head & Swicegood, 2013). Web probing is an alternative approach which does not occur in real-time as the procedure takes place in an online survey (e.g. Lenzer & Nuert, 2017).

Figure 1: Approaches to conducting cognitive interviews

‘That Old Pair of Jeans’: Phone

The general consensus from existing research is that phones are only appropriate under specific scenarios, such as when there are cost constraints, text only material needs to be tested and the target population is time constrained, such as clinicians (Noel, 2013).  However, when the disadvantages are considered, there appears to be little value in using phones:

  • The inability to capture non-verbal cues presents a significant challenge for understanding participants’ reactions. A period of silence could be interpreted in multiple ways by relying on audio alone, such as confusion or boredom, whereas monitoring facial reactions would help to distinguish them apart (Noel, 2013).
  • Sustaining engagement with participants is problematic, while limiting the duration of interviews to a shorter period of time, such as 30 minutes, would require fewer questions to be asked and more interviews to take place.
  • The setting where the respondent conducts the interview can be inappropriate which may have a negative impact on the quality of the data.

‘(Telecommunication) Machines Can Do The Work’: Web-Based Video Conferencing

Web-based video conferencing, such as Skype and Zoom, represents the most appropriate alternative for interviewers who want to carry out cognitive interviews using real-time communication:

  • ‘More nuanced findings’ can be produced as participants’ reactions and non-verbal cues to questions are observable, such as indications of confusion or distress (Dean, Head & Swicegood, 2013). However, the quality of data still tends to be poorer compared with in-person participation.
  • Recording features, which are commonly embedded into tools and applications, capture footage that can be used for post-interview analysis, including the amount time spent on each item and interaction with the instrument.
  • In many cases, data collection will take part in participants’ own homes which raises issues about privacy and confidentiality that will need to be addressed in the research design (Ravitch, 2020).
  • One of the main challenges with using this approach, compared to in-person delivery, is that more interruptions are anticipated, such as distractions that occur off-camera (see Figure 2).

Figure 2: Off-camera distractions (BBC News, 2017)

‘Slash Dot Dash’: Web Probing

The approach of web probing occurs without the involvement of an interviewer and the data collection takes place in a self-administered form (Meitinger & Behr, 2016; also see Kaczmirek, 2017, Figure 3). Participants respond to each survey question before answering follow-up probing questions, which are usually located on a subsequent page. There are a number of differences between cognitive interviewing and web probing, including:

  • Web probing is restricted to probes that have been pre-determined, while cognitive interviewing allows participants to be asked follow-up questions that are spontaneous and adaptable. The increased interaction of cognitive interviewing is thought to generate more relevant responses and more errors in surveys being detected (Meitinger & Behr, 2016), although Fowler and Willis (2020) found both approaches generate similar information.
  • In web probing, it is not possible to capture articulations from the ‘think aloud’ technique, while non-responses (for example, where responses consist of no words or are incomprehensible) are more frequent and larger sample sizes are required (Meitinger & Behr, 2016). A tool has been developed by Kaczmirek, Meitinger and Behr (2017) to help users detect different types of non-responses to probing questions.
  • The use of multiple cognitive interviewers could lead to a lack of comparability across interviews if there are any variations in practice (Beatty & Willis, 2007).
  • Web probing allows participants to take part at a time that suits them but there is greater risk that instructions might be misunderstood and there is no opportunity for them to receive clarification about tasks.
  • The data does not need to be transcribed when web probing is used as it is immediately stored in an electronic format.

Figure 3: Assessing survey items online with web probes (Kaczmirek, 2017)

Meitinger and Behr (2016) suggest using a combination of cognitive interviewing and web probing within the same study.  Cognitive interviewing is deemed to be most suitable at the pre-testing phase while web probing can be adopted during data collection (for example, to measure quality control) or after data collection (for example, to aid the interpretation of surprising findings).

‘Praise You’: Respecting Participants and Next Steps

Once the research design has been finalised and the most appropriate method for your own context has been determined, ‘approach study participants with respect, humility, and appreciation for their time’ (Ravitch, 2020). Research should take place around the needs and schedules of participants.  In terms of next steps in the project at Sheffield Hallam, a decision was made to postpone the cognitive interviews with students until in-person participation can take place.  However, the exploration of alternative ideas has provided a number of options that might be amalgamated into our plans in the future.

Like many across the sector, I’ve been working from home in recent weeks. Music has acted as a form of escapism.  Congratulations to any readers that have sussed out that the sub-headings relate to songs released by Fatboy Slim, who has featured prominently on my recent playlists. 


Austen, L. (2020, March 23). Covid-19: SHU guidance for staff and students undertaking institutional research and evaluation [Blog post]. Retrieved from

BBC News. (2017, March 10). Children interrupt BBC News interview [Video file]. Retrieved from

Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public opinion quarterly71(2), 287-311.

Behr, D., Meitinger, K., Braun, M. & Kaczmirek, L. (2017). Web probing – implementing probing techniques from cognitive interviewing in web surveys with the goal to assess the validity of survey questions. Mannheim, GESIS – Leibniz-lnstitute for the Social Sciences (GESIS – Survey Guidelines).

Blair, J., & Conrad, F. G. (2011). Sample size for cognitive interview pretesting. Public opinion quarterly75(4), 636-658.

Dean, E., Head, B., & Swicegood, J. (2013). Virtual cognitive interviewing using skype and second life. Social Media, Sociality, and Survey Research, 107-132.

Fowler, S., & B. Willis, G. (2020). The practice of cognitive interviewing through web probing. Advances in Questionnaire Design, Development, Evaluation and Testing, 451-469.

Kaczmirek, L., Meitinger, K., & Behr, D. (2017). Higher data quality in web probing with EvalAnswer: a tool for identifying and reducing nonresponse in open-ended questions.

Kaczmirek, L., Meitinger, K., & Behr, D. [Lars Kaczmirek]. (2017, June 2). Web probing & EvalAnswer: Assessing survey items online with web probes and reducing nonresponse [Video file]. Retrieved from

Lenzner, T., & Neuert, C. E. (2017). Pretesting Survey Questions Via Web Probing–Does it Produce Similar Results to Face-to-Face Cognitive Interviewing?. Survey Practice10(4).

Lupton, D. (2020). Doing Fieldwork In A Pandemic. Retrieved from

Meitinger, K., & Behr, D. (2016). Comparing cognitive interviewing and online probing: Do they find similar results?. Field Methods28(4), 363-380.

Noel, H. (2013, May). Conducting cognitive interviews over the phone: Benefits and challenges. In 68th Annual Conference of the American Association for Public Opinion Research (pp. 16-19).

Ravitch, S. (2020). The Best Laid Plans…Qualitative Research Design During COVID-19. MethodSpace. SAGE publishing. Available at:

Sheffield Hallam University. (2020). Guidance and Legislation. Retrieved from:

Willis, G. B. (2004). Cognitive interviewing: A tool for improving questionnaire design. SAGE publications.