FAQs: Analysing WOMBAT data

Q: HOW DO I ANALYSE WOMBAT DATA?

The WOMBAT Data Analysis Guide provides a guide to WOMBAT data analysis. If you need further assistance, we recommend seeking advice from a qualified statistician. The following reference may also be of assistance:

• Walter SR, Dunsmuir WTM, and Westbrook JI. (2019) Inter-observer agreement and reliability assessment for observational studies of clinical work. J Biomed Inform 100: 103317.

Q: HOW DO I CALCULATE IRR?

Inter-rater reliability (IRR) assessment is a fundamental aspect of data quality when there are multiple observers. Assuming you have performed one or more sessions where multiple observers have simultaneously, but independently, followed the same participant, then assessing IRR for WOMBAT data has some unique considerations. Since tasks time stamps rarely agree exactly between parallel observers, assessing IRR at task-level is very difficult. Instead it is better to convert the data to small time windows, where window width is defined by the smallest time unit. For example, if time stamps are recorded to the nearest second, then one second windows should be used. Univariate or multivariate chance-adjusted agreement measures can then be applied to the time window data to generate time-based IRR estimates. Where multiple variables are to be used in the analysis, assessing multivariate agreement is recommended. This can be done with the iota score for nominal variables [1], or under some conditions the mean of univariate Cohen’s kappas can be used. Relevant details are discussed in Walter et al. [2] section 12.3.

[1] Janson H, Olsson U (2001) A measure of agreement for interval or nominal multivariate observations. Educational and Psychological Measurement 61(2): 277-289.

[2] Walter SR, Dunsmuir WTM and Westbrook JI. (2019) Inter-observer agreement and reliability assessment for observational studies of clinical work. J Biomed Inform 100: 103317.

Q: CAN I USE HYPOTHESIS TESTS (E.G. T-TESTS) ON WOMBAT DATA?

The type of analysis methods depends on the study design and the research questions, but in general there are several caveats with using conventional hypothesis tests. The most important consideration is that WOMBAT studies tend to be observational and hypothesis tests applied to such data ignores confounding factors and can then produce biased results. Such tests should only be applied if the design allows, as when confounding factors are controlled by some means. Otherwise multivariate methods should be used instead of simple hypothesis testing. If testing is appropriate in your context then consider Monte Carlo testing as this avoids distributional assumptions which are often not satisfied by WOMBAT data.

Q: WHAT PROGRAM SHOULD I USE TO ANALYSE WOMBAT DATA?

The choice of software for analysing WOMBAT data is essentially up to the researcher. Ideally a program that allows custom manipulation of data through written commands is preferable if the researcher has skills in using one of those, for example R, SAS, Stata or SPSS. It is possible to use spreadsheet programs such as Excel, but this tends to be time consuming and error prone.

A set of SAS macro programs, and accompanying guidance documentation, are that will calculate summary statistics with confidence intervals. These can be implemented by anyone with basic SAS skills. If you would like a copy of the SAS macro programs, please contact the WOMBAT team via email: wombat@mq.edu.au.