Today’s Driver Monitoring Systems focus heavily on eye movements, but they miss an important dimension of human behavior; hand movements. Integrating hand monitoring alongside posture and gestures offers a richer, more accurate understanding of driver distraction, fatigue, stress and intentions. This approach enables vehicles to respond more intelligently and in real time, marking a significant step toward safer and more intuitive driver understanding.
Many new cars on the market today are equipped with a driver monitoring system which analyses the driver’s eye movements to detect fatigue and distraction. These help the vehicle understand where the driver’s visual attention is directed, but there are other behavioral cues that can help paint the full picture of the drivers’ awareness and intentions. Driver hand movements and posture is a less researched field than eye-tracking, but studies indicate that these are useful for the holistic understanding of the driver’s current state with regards to distraction, drowsiness and stress.
Detecting Distracted Driving
Approximately 1.16 million people die each year as a result of road traffic crashes, many attributed to distracted driving. Detecting driver distraction, in order for the vehicle to issue a warning, is often done by monitoring the driver’s eye glance behavior using an in-cabin camera.
One of the most important studies in the field of driver distraction is NHTSAs 100 car study. The study collected comprehensive data on driver behavior from 100 vehicles over a year and has been referenced in the foundation for much of today’s regulatory landscape for driver monitoring. It found that a majority of crashes and near-crashes which occurred had an inattentive driver, and the most frequently observed source of inattention was engagement in a secondary task unrelated to driving.
Specifically moderate to complex secondary tasks significantly increased the risk of a crash, such as reaching for an object or using a mobile phone, whereas simple secondary tasks which require, at most, one button press or glance from the forward road had no noticeable impact on crash risk. Looking at the data from crashes occurring in the study, only 25% of the cases had the driver holding both hands on the wheel.
Cognitive State
Hand monitoring can also give insights to the cognitive state of the driver. When studying how frequently drivers subconsciously touch their face while driving - research performed during the global covid pandemic - researchers found a link between how often the drivers would touch their face and how demanding the current driving situation was. In non-demanding driving situations, the drivers would on average touch their face 26 times per hour, whereas in more demanding driving situations, such as frequent lane change, this number went down to just above 4 events per hour, likely connected to a need for more active control of the vehicle.
In situations outside driving, increases in the occurrence of spontaneous touching of the face has been linked to stress during cognitive work and refocusing of attention to a cognitive task after a distraction . While these behavior patterns have not been verified in manual driving conditions, they could help build an understanding of the driver’s cognitive state during hands-free automated driving, thus aiding the understanding of driver readiness to take over the wheel. The combined findings, while initially seeming to counter prove each other, could also potentially be useful for detecting cases of mind-wandering behind the wheel as drivers seem less likely to engage in spontaneous face touching behavior when fully focused on the driving task.
Understanding Driver and Passenger Intentions
For automated driving systems, understanding the driver’s intentions is important. Some automated systems allow the driver to give their own steering input while the system is active, so called collaborative steering. Knowing at an early stage that the driver plans to give their own steering input could be used to make the steering control smoother.
A study observing drivers’ behaviour before and during a turn at an intersection found that the driver’s intention to turn could be detected a second before the actual turn through observing their hand movement patterns. Independently observing head movements also gave similar results. But by combining the hand-tracking cues with analysis of the driver’s visual scanning behavior the prediction was possible already two seconds ahead of an event. Beyond prediction, these intended driver interventions could additionally be differentiated from unintended steering input due to engagement in a non-driving task, such as reaching for an object in the car.
For HMI development, knowing that the driver is about to use the infotainment screen can be used to develop smarter, less distracting interfaces. Selection can be made faster and easier by presenting the most relevant information for the situation or enlarging the most commonly used buttons as soon as the driver reaches for it, before any touch screen input is given. Understanding if it’s the driver or the passenger who are performing the interaction is also crucial information for tailoring the infotainment experience. Passengers can be allowed to perform more tasks on the touch screen than driver while the car is in motion, ensuring both safety and convenience.
The Road Ahead
Incorporating hand monitoring into existing driver monitoring systems represents a significant leap forward in accurately assessing driver state. By combining eye tracking with hand movements, posture and behavioral patterns, vehicles can achieve a far more nuanced understanding of distraction, cognitive load, stress and driver intent. As automated driving technologies advance, this holistic approach will be essential for ensuring smoother human-machine collaboration, reducing distraction-related risks, and creating safer, more intuitive driving experiences.
Neonode's MultiSensing platform is capable of finding and tracking hands in real time, based on the images from a single in-cabin camera. The software can be used to analyze hand-movements, see what the driver and passengers are interacting with as well as classify if a driver is holding a non-driving related object. The system also features a specific detection for hands-on wheel status, built to detect the actual grip of the steering wheel. The different hand-tracking functionalities can be utilized to optimize distraction warnings, both in manual and automated driving scenarios.
U.S. Department of Transportation NHTSA (2010), ‘An Analysis of Driver Inattention Using a May 2010 Case-Crossover Approach On 100-Car Data: Final Report’ Ralph, F. et al. (2022), ‘U can’t touch this! Face touching behaviour whilst driving: implications for health, hygiene and human factors’, Ergonomics, 65(7), pp. 943–959. University of Houston (2025), Touching Your Face May Reveal Hidden Stress, University of Houston Study Finds | University of Houston, date acquired 2026-07-01 Mueller SM, Martin S, Grunwald M (2019), ‘Self-touch: Contact durations and point of touch of spontaneous facial self-touches differ depending on cognitive and emotional load’, PLoS ONE14(3): e0213677 |Shinko Yuanhsien Cheng and Mohan Manubhai Trivedi (2006), ‘Turn-Intent Analysis Using Body Pose for Intelligent Driver Assistance’, IEEE Pervasive Computing, vol. 5, no. 4, pp. 28-37 Eshed Ohn-Bar, Ashish Tawari, Sujitha Martin, and Mohan M. Trivedi (2014), ‘Predicting Driver Maneuvers by Learning Holistic Features’, IEEE Intelligent Vehicles Symposium Proceedings, Dearborn, MI, USA, 2014, pp. 719-724