Characterising human cognitive processes through behavioural and physiological analysis

Centre for health informatics

Research stream

Precision Health

Project members

Ronie Taib - PhD candidate and Principal Research Engineer (CSIRO)
Shlomo Berkovsky – Professor Precision Health
Roneel Sharan – Research Fellow

Project contact

Ronnie Taib
E: ronnie.taib@students.mq.edu.au

Project main description

Wouldn’t it be nice to adapt educational material to the cognitive capacity of a learner, even as it fluctuates in a day due to their environment, fatigue or psychological state? We know a lot about how the mind works, yet we lack the multi-disciplinary tools to measure its performance in relation to the tasks at hand. This research project utilises unobtrusive, real-time physiological and behavioural sensors and artificial intelligence (AI) methods to correlate how people do and react to things, to what mental state they are in.

In health informatics, this promises to help practitioners objectively manage their own performance during learning and on the job. By monitoring the physiology and behaviour of the health professional, a system could alert them that their latest decision may not be as confident as usual, although they may not have consciously realised it. On the patient side, we showed that such a system could help determine the personality traits of a person watching short videos and sets of photos, replacing traditionally long questionnaires.

This research brings together medicine, psychology, signal processing, data analytics and machine learning. It is based on user studies in controlled conditions, feature sensors such as eye tracking, skin conductance (related to sweating), electroencephalogram (EEG), as well as simple interactions such as mouse movement.

If you would like to partner with us on developing this project further, please contact ronnie.taib@students.mq.edu.au

References

1. Ronnie Taib and Shlomo Berkovsky (2020). Modeling humans via physiological and behavioral signals. Interactions, 27(3), 30-34. https://doi.org/10.1145/3386237.
2. Shlomo Berkovsky et al. “Detecting Personality Traits Using Eye-Tracking Data”. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. CHI ’19. New York, NY, USA: ACM, 2019, 221:1–221:12 [Best paper award]
3. Ronnie Taib, Jeremy Tederry, and Benjamin Itzstein. “Quantifying driver frustration to improve road safety”. In: CHI ’14 Extended Abstracts on Human Factors in Computing Systems. Toronto, Ontario, Canada: ACM, 2014, pp. 1777–1782
4. Kun Yu et al. “Mouse Behavior as an Index of Phishing Awareness”. In: Human Computer Interaction – INTERACT 2019. Ed. by David Lamas et al. Lecture Notes in Computer Science. Springer International Publishing, 2019

Project sponsors

Macquarie University, Data61-CSIRO

Project status

Current

Centres related to this project

Centre for Health Informatics