Views from the Field No. 5 | Improving data accuracy and consistency using interviewer instructions

Tweet

 By Callum Taylor

From our experience, one of the easiest and most effective ways to ensure the consistency of data collected during CAPI surveys is through the use of interviewer instructions that are programmed into the surveying software and are visible to interviewers during surveys. Such instructions can provide a number of benefits:

• Guiding interviewers through the survey – This helps to ensure that all interviewers navigate through the survey in the intended way. Using such instructions together with in-built enablements and routing patterns can help improve the flow of questions asked to the respondents and provide clear and early communication to the interviewer about the next steps they should be taking to progress through the questionnaire in the dynamic context of the respondents live answers. This is important as the order that questions or sections appear in can directly impact the responses given. The most common reason this happens is when respondents provide an unintended response. For example, answering questions regarding your opinion of certain government policies could later bias your response to a question on your opinion on the government or certain politicians if you base that response solely in relation to the policies previously asked about.

• Providing instructions on how to complete activities that don’t involve asking face-to-face questions – For example, reminding interviewers of the respondent selection protocol, or the methodology for a random walk, instructions for taking photos of certain items (to prove ownership or assess cleanliness), or instructions for asking interviewers whether they have observed certain household characteristics or items.

• Providing instructions on how to ask certain sensitive, or complicated questions – In the case of sensitive questions, this can help to ensure the respondent feels comfortable and secure answering such questions with each interviewer, thereby lowering the probability of refusals and missing data. Instructions can also help both the interviewer and respondent understand how to ask and answer complicated questions. This helps to ensure that the question is understood clearly and answered properly, providing clean and consistent data of high quality.

• Providing instructions on how to record answers – For example, by providing instructions on the units of measurement. This helps to ensure that all interviewers record answers in the same way.

• Providing instructions on what to do in certain situations – This can be useful for a number of reasons, for example by giving instructions to interviewers on what to do if no eligible respondent is available, or if a respondent refuses to take part in the questionnaire, or if a replacement household or respondent is required. Such instructions can help to ensure that interviewers adhere to the agreed protocols, which can be particularly important for selection of respondents and replacements.

• Quality control checks – Instructions can also be used as a form of quality control, for example by programming an instruction which indicates to the interviewer or data user which interview file they are using, or programming an instruction to appear when a certain answer is given to a question. This provides instant feedback to the interviewer, instead of them having to validate the question once they arrive at the end of a screen or interview file.

• Use of instructions as ‘gateways’ – A dynamic instruction could be linked to an interviewer only question to ask for clarification on a topic which, before they can proceed, commits the enumerator to making a statement of progress or action completed. At the basic level this acts as a prompt or reminder but, in a key section, commits them to an answer so that enumerators cant ‘evolve’ their flow through the questionnaire as they become more familiar with it to manipulate the questions to be asked, the responses or re-edit them afterwards.

The easy-to-use interface of the surveybe Designer allows users to quickly and easily programme instructions within a survey. Moreover, these instructions can be customised with different fonts, text colours and sizes, or programmed to dynamically appear in certain situations. The surveybe Designer allows interviewer instructions to be utilised in different ways as a part of different questionnaire components:

1. Within question texts – Using HTML code to alter the formatting of text, for example to assign different colours to different parts of the question text which allows the user to provide instructions to the interviewer about how to ask and answer the question correctly. The use of different colours indicates to the interviewer which sections of the text are intended to be read to the respondent, and which are instructions.

Figure 1 – Programming interviewer instructions within question texts

Useful HTML for surveybe

2. Using labels – As well as within ‘question texts’ surveybe allows the user to insert labels in their questionnaire. These are clearly displayed in the questionnaire file. Labels can be used for a variety of functions, but one of the most beneficial is in giving instructions to interviewers. Labels can be customised using HTML to display different colours and sizes of text, and with SQL to display different texts in different circumstances. Surveybe also allows users to embed media files and images into labels. One use for this would be for tracking purposes, for example having a label displaying a photo of the respondent from a baseline survey alongside a question asking the interviewer to confirm they have located the correct respondent for the follow-up. Further details and examples of such uses can be found in the EDI blog ‘Keep successfully on track: Utilising tracking tools in baseline surveys‘. Examples of dynamic labels are given below.

3. Using documentation – Surveybe also allows users to create documentation for every question within a survey file. This documentation can be viewed by interviewers whilst they are conducting a survey and this is therefore another useful way of providing important information to interviewers to ensure that the data collected is accurate and consistent. Documentation fields can be attached to all screens, rosters and questions within the questionnaire and can be used to make notes about a particular section or question. It could be used, for example, to define terms within the question, or to provide instructions on when to select each of the different response options.

Figure 2 – Adding documentation to a question

It is worth noting that both question texts and labels (and hence any interviewer instructions within them) have multi-language functionality. By using language sheets and reference tables, surveybe can be easily programmed to display a number of different languages, depending on the fieldwork requirements. This ensures that any instructions are fully understood by the interviewing teams.

Examples of these uses are detailed below:

Basic instructions about conducting the survey

Example 1: Using coloured interview instructions to have the interviewer check they are using the correct interview file for the facility or household they are interviewing. Using SQL and reference tables, such labels can be programmed to dynamically update and display the correct information for each observation in a sample. This helps to ensure that the data collected is done for the correct observations within a sample.

Figure 3 – Instructions for checking the correct file is being used

Example 2: Using coloured interview instruction to provide useful information about the respondent or the questionnaire in a label. This can help improve the efficiency of the survey and improve the consistency of data collected.

Figure 4 – Instructions to provide useful information to the interviewer

Instructions to enforce quality control checks
Example 3: Using coloured instructions in a label to inform the interviewer of a response that needs to be checked or re-entered. This gives immediate feedback to interviewers and can help to avoid errors being made.

Figure 5 – Using instructions for quality control and data validation

Example 4: Instructions informing the interviewer what to do when certain situations arise during the survey. This can help to ensure that interviewers follow the agreed-upon protocol, which in turn can improve the consistency of data.

Figure 6 –  Using dynamic instructions to ensure an eligible respondent is selected

Figure 7 – Using dynamic instructions for different responses to the same question (1)

Figure 8 – Using dynamic instructions for different responses to the same question (2)

Instructions such as the ones displayed above are a useful way of allowing the researcher to communicate remotely with interviewers in the field. They enable the researcher’s voice to be heard by all interviewers, at all times, reminding them of the research expectations. Furthermore, surveybe also allows interviewers to add comments to all questions within a survey, alongside the actual answers, making the conversation between researcher and interviewer a two-way process. Analysis of these comments over time can give valuable insights into ways in which the questionnaire design could be improved, or could indicate that the instructions aren’t being understood as clearly as the researcher may have expected.

For example, an interviewer comment on a recent survey alerted researchers to the possibility that the death of a household member could occur between different rounds of a survey. This allowed the researcher to modify the survey so that questions about any deceased persons were not asked to the respondent. Early identification of this issue avoided unintentional distress being placed on other respondents, which ultimately could have affected the quality of data.

In a previous blog I wrote about the importance of interviewer training, and how we at EDI take very seriously our commitment to training and up-skilling our staff. While interviewer instructions should never be viewed as a replacement for full and effective training of interviewers, they can be a useful tool for reinforcing the lessons learned from training and the wider field protocols. Using such instructions in combination with training, and a well-designed survey can help greatly improve the accuracy and consistency of data collected.