What do I need to consider before starting a WOMBAT study?

Which study data collection template – existing or new?

As with any research project, there are a number of key elements that need to be considered when developing the protocol (research plan) for a WOMBAT study.

Consider whether the research question(s) you wish to answer require the development of a new study template, or whether a study template used in a previous WOMBAT study could be used. Developing a new study template can take time and several iterations to ensure all activities of interest are sufficiently captured by the task classification underpinning the template. The task classification (i.e., the definition/scope of each variable included in a study template) needs to be clearly described in the study protocol.

Whenever possible, we suggest using a previously validated study template and/or task classification. If a previously developed study template does not exactly fit your study needs, it may be easy to make some modifications to the template rather than creating a new one. Using a previously developed template should speed up the design process and enable quicker piloting, observer training, and progression to data collection.

Designing a new study template

You need to consider the number of variables (dimensions, categories and subcategories) that you will include. In WOMBAT, it is possible to add as many dimensions and categories as you wish. Bear in mind, however, that data collection will be difficult if there are too many categories, and this may also require observers to scroll the screen and inhibit accurate data collection. Attempting to collect very detailed information will also prove difficult if observers cannot keep up with the speed at which activities change.

Study template designers also need to think about:

• Which dimensions should be mandatory? The first dimension on any study template is the study focus and is automatically mandatory. Other dimensions which are important to the research question should also be mandatory.
• When should dimensions be optional? Optional dimensions can make data collection easier for observers. If, for example, you are only interested in collecting information on computer use for certain tasks then if these tasks are not being undertaken, “computer” does not need to be selected.
• Which dimensions should allow multiple selections? Multiple options and/or selections should be used for dimensions which aren’t the direct focus of the study.

Selecting and training observers

The quality of your WOMBAT data will depend upon the quality of the data collection processes established. Observers need to undergo training and consistency between observers should be tested prior to commencing formal data collection.

As the primary data collectors, observers are integral to the success of any observational study. Whilst it may be desirable to select observers that already have an intimate knowledge of the work environment being studied, this is not always possible. Thus, training is key and training modules should include:

• Outlining the purpose of the study, background, method and overview of protocol;
• Providing detailed explanations of definitions used for the study template and task classification, with discussion of examples;
• Instructions on WOMBAT App use (see the WOMBAT App training videos in the WOMBAT Resources section) and further explanation of study template definitions, with practice scenarios;
• Short simultaneous observation sessions between the teaching/experienced observer and trainee observer(s), to ensure the trainee has understood the task classification and is applying it correctly when recording activities in the study template;
• Assessment of inter-rater reliability, with both the experienced and trainee observer simultaneously, but independently, carrying out a thirty-minute period of observation on the same participant. The trainee observer is deemed to be able to carry out independent observational work when inter-rater reliability, as measured by kappa scores, is equal to or greater than 0.85.

The above is as a very general guide. The time it takes to train observers will be highly dependent upon the complexity of the study and the data collected using WOMBAT. If interruptions are involved in the project, the assessment of inter-rater reliability may involve checking the type and frequency of interrupting and interrupted tasks. For those projects with multitasking, the length of time on multitasking should also be examined.

Observation schedule

With the WOMBAT study research questions in mind, consideration needs to be given to determining the observation period (e.g. day/evening/night shifts, weekdays, weekends, public holidays). Observation periods should be equally distributed, and observation of participants should be randomised across the selected times/days/observers. In most previous WOMBAT studies, observation periods were equally distributed across between 0700 to 1900 hours, Monday to Friday. Public holidays were not included. Each observation period was a maximum of two hours, and no individual participant was observed for more than a total of ten hours.

The length of each observation session also needs to be considered. Depending on the complexity of the study data collection template and the activities being observed, each observer may only be able to observe for a total of 1 to 2 hours at a time. After this time period, researcher fatigue may impact on the data quality.

Ethics

Ethical approval from an institutional Human Research Ethics Committee may be required prior to commencing data collection. Approval from study sites may also be required.

The principles of ethical research demand that all identifying details obtained by researchers remain confidential. We recommend de-identifying data relating to participants in the following ways:

• Assigning a unique number to each participant at the time of written consent;
• Using only the unique number for data collection purposes;
• Undertaking analysis using only the unique number;
• Keeping any documents linking unique numbers to participant names separate from collected data, and stored in locked cabinets or on password protected computers;
• Limiting access to data to the researchers involved in data collection and analysis;
• Presenting all data in aggregate form.