Basic Study for Grasping the Human Gaze Movement of a Remoting-Ship Operator

Our research focuses on understanding the disparity in risk perception when operating maritime autonomous surface ships (MASS) between onboard and through screens. In our initial step, we conducted experiments using a ship simulator, involving 4 students aspiring to be navigators. They navigated through screens and participated in debriefing sessions. Human gaze movement was utilized to ascertain the ability to comprehend navigational risks solely through observation. Simultaneously, salivary alpha-amylase samples were taken to verify their accurate risk cognition. The key finding indicates that human gaze movement aids in precisely understanding navigational risks, suggesting the feasibility of future ship operations through screens. Our research targets navigators crucial in global logistics, aiming to enhance safety in remote ship operations by demonstrating that risk cognition is achievable through human gaze movement. This breakthrough is vital for the evolving realm of MASS operations.

Introduction

In the hope of reducing maritime accidents caused by human error and improving the working environment for seafarers, the development of maritime autonomous surface ships (MASS) is underway in all countries of the world. MASS is intended to reach full autonomy or become a remotely-controlled unmanned ship, and the actualization of MASS operation has been discussed as early as 1964 by the Maritime Safety Committee (MSC) under the International Maritime Organization (IMO). Recently the realization of MASS operations is rising, but no specific legislation has been laid down for them. The 105^th MSC in 2021 made an arrangement for formulating international rules for MASS without mandatory and developed a roadmap showing the work plan while looking at future mandatory of these rules. Finally, the 106^th MSC in 2022 decided to establish the MASS operation rules based on the suggestion of Japan, which leads the realization of MASS operations throughout the world.

In Japan, the MASS designed by the Ministry of Land, Infrastructure, Transport and Tourism (MLIT) is expected to be put into practical use by 2025 for the purpose of reducing the burden on the current seafarers’ work by The Nippon Foundation (2022). Recent maritime research in Japan has mainly developed the practical application of new vessels under the leadership of the MLIT, and The Nippon Foundation (2020) which puts a project called “MEGURI2040”.”In MEGURI2040, the MASS designed by the Nippon Foundation for the realization of legal development proposals to the IMO will be put into practical use by 2040 with the assumption that 50% of domestic operating vessels will be replaced by MASS. For the automatic avoidance of other vessels, MASS is required to utilize technologies such as Artificial Intelligence (AI) and Information and Communication Technology (ICT) on the premise that operators will participate in the operation. This foundation’s try, MEGURI2040, is expected to reduce marine accidents resulting from human error, which is estimated to be 70%–80% of the marine accident factors.

The MASS navigation system utilizing AI/ICT technology is designed to remotely control a ship through screens.

Currently, legal development is underway for the leadership of the IMO; with the completion of MASS, we will be faced with the question that who will be responsible for steering it. In view of this legal development, we think that minimizing the difference between the risk cognition by navigators at the ship bridge (bridge) in maneuvering a ship and by the operators in the same remote-maneuvering ship would be an issue which arises when these new MASS turn up. To give one specific example of the issue, when comparing the views from a bridge and the scenery through the screen, a small ship near the own ship may be hidden in the viewing angle. To begin with, we consider that it is possible to grasp the depth of the actual scene through the screen.

Therefore, the primary objective of this research is to substantiate distinctions in the identification of other vessels, comparing perspectives from the actual ship maneuvering environment on the bridge with those observed through a screen. This paper initiates with a presentation of the findings from an experiment designed to investigate whether the subjects of new navigational officers would discern potential collision risks with other vessels through a screen.

In this research experiment, we will use alpha-amylase to confirm the degree to which subjects are aware of the risk of collision with other ships while navigating in a ship simulator scenario. Simultaneously, we will use an eye-tracking device to determine the subjects’ ability to focus on appropriate risk sea areas within their line of sight. This dual approach aims to validate whether ship operators can effectively recognize risks when operating a ship through a screen.

Human Gaze Movement

There are various methods to measure visual field and line of sight (Krafkaet al. (2016), Rayner (1998), Von Zezschwitzet al. (2013), Wedel and Pieters (2007), Zhanget al. (2018)). Human gaze movement tracking technology is widely utilized as an indispensable tool in contemporary research due to its high precision and non-intrusive nature. This technology captures participants’ eye movements, providing profound insights across various research fields. While there are various points of focus in other eye research than blinking, this research uses gaze movement, which simply follows the line of sight. This Gaze movement is measured where you are looking from the direction of your eyeballs. In this study, we will verify where the subject is looking at the object. The state of gazing at something and keeping your gaze fixed on it is called “mooring.” In this study, we examine the timing at which subjects perceive risk by understanding the mooring that can be obtained from the measurement time of where they are looking.

Salivary Alpha-Amylase

Autonomic and endocrine nervous systems in human bodies are often used as indicators to measure physiological responses to stressors as represented by Wehrweinet al. (2016), Rohlederet al. (2004) and Ulrich-Lai and Herman (2009). Preceding studies by Sluiteret al. (2000) and Goldstein (2003) used and evaluated mental stress hormones such as cortisol and catecholamines in the blood, however, these preceding methods required blood sampling, which is a problem because the sampling itself became a stressor and does not allow for accurate stress evaluations. On the other hand, we consider the measurement of substances in saliva to is a more accurate method of evaluating human stress than blood because saliva sampling is not painful physically and mentally. In this study, we adopt salivary alpha-amylase activity in saliva for the objective evaluation of subjects’ mental stress in maneuvering the ship under a ship navigation simulation; alpha-amylase activity is more responsive to mental stress than salivary cortisol and can be said to be sensitive to the changes in sympathetic nerve activity by Nater and Rohleder (2009).

Purpose

To understand the degree of recognizing ships based on gaze movement, this research uses salivary alpha-amylase as a comparison indicator, which has been used to accurately measure the degree of human tension in previous research. After that, we verify the effectiveness of gaze movement to grasp navigators’ mental workload for the risk of colliding with other ships.

Materials and Method

Participants

The subjects in this research comprised 4 young male navigators A-D, all aged 20, with aspirations to attain the rank of third-class marine engineer. Each subject engaged in a ship simulator scenario, simulating a single voyage.

Equipment Tracking Human Gaze Movement

EyeTracking Core + (SiB Co.,Ltd.) was used to measure eye movements. This device determines the direction of the line of sight based on the diameter of the pupil and coordinate values relative to the device, and also determines eye movement from information related to blinking, and determines whether it is mooring or not. The computer screen displays the subject’s line of sight in red, reflecting the direction of the subject’s visual focus. The EyeTracking Core + system records the visibility conditions during measurements, allowing for the tracking of the subject’s gaze locations while maneuvering the vessel by following the red cross like on the upper center screen in Fig. 1. The device required prior to calibration, which was done before each experiment.

Fig. 1. The red cross shows human gaze movement.

Device Measuring Salivary Amylase

This research adopts Nipro’s α-amylase kit, which consists of a disposable sheet for collecting saliva and a device that calculates salivary amylase from the sheet and displays the value. The chip consists of a holder and a sheet (Fig. 2). The tip of the sheet is placed under the tongue for 30 seconds to collect saliva, and the tip of the sheet is placed inside the holder, then the chip is placed in the device. You can get a value which is shown by the device. Values range from 10-200kIU/L. The device is 130 mm × 40 mm × 87 mm, 190 g. The tip is 13 mm × 6 mm × 120 mm, 3 g.

Fig. 2. The chip of Nipro’s α-amylase kit.

Ship Navigation Simulator

This study utilized the ship simulator available at the National Institute of Technology, Toba College (Figs. 1 and 3).

Fig. 3. Ship navigation simulator through screens.

This ship maneuvering simulator is usually used to conduct ship maneuvering training on the bridge, which simulates an actual ship maneuvering environment with a steering wheel and a compass. However, since this time subjects will maneuver the ship through a monitor, in this experiment, subjects operated the own ship in a specific room where we usually control the movements of other ships and can create scenarios.

To facilitate a clear recollection of the subjects’ cognitive processes while maneuvering the ship during the post-experiment debriefing, we constructed a scenario in which there was a high probability of an immediate collision, regardless of the ship’s maneuvering. The virtual own ship’s performance is shown at both sides of the pilot card in Figs. 4 and 5.

Fig. 4. The face of the pilot card.

Fig. 5. The back of the pilot card.

The distance and course of both the own ship and other vessels within a one-minute interval in the initial settings are presented as vectors (Fig. 6). The vector representing the own ship is blue, while those representing other ships are yellow. Importantly, no external forces, such as tides or wind, influenced the observations during this scenario. In this experiment, we decided not to control the movements of other ships with the initial settings as shown in Fig. 6.

Fig. 6. The scenario of the ship navigator simulator.

Own ship’s speed started ‘half ahead’. The engine motion and steering allowed free control. We made subjects to get the information necessary for navigation instantly from the start of the experiment, without having time to learn information about other ships, such as speed and course, from electronic charts and thinking about what kind of ships to avoid there are. The experiment started from a sight where subjects recognized the same view obtained when standing on the bridge of the ship. During the experiment, the subjects were allowed to use binoculars sight to enlarge the image and to change the field of view. Fig. 7 shows the experimental situation where saliva is sampled at one-minute intervals to measure salivary alpha-amylase.

Fig. 7. The actual experimental situation.

Results

Figs. 8–10 show the results of alpha-amylase concentration fluctuations sampled at one-minute intervals from the start of the experiment. As a result, all of the subjects collided with other vessels within 4 minutes of starting the experiment. It is difficult and gave up to uniformly compare alpha-amylase results because each subject differed in how they maneuvered the ship. Therefore, we confirm the results of each Figs. 8–10 through the effectiveness of alpha-amylase by asking each person after the experiment, “What were you thinking at that time?” and confirm whether the line of sight at that time was properly aware of the risks of other ships.

Fig. 8. The result of alpha-amylase concentration fluctuation for subjects A and B.

Fig. 9. The result of alpha-amylase concentration fluctuation for subject C.

Fig. 10. The result of alpha-amylase concentration fluctuation for subject D.

From the results of debriefing and gaze movement, subject A visually recognized the other ships involved in a collision at 1′30′′ and 2′50′′ after the start of this experiment, and the own ship navigated by subject A collided to the one other ship at 3′30′′. Subject B saw a ship that could potentially collide at 0′40′′, and the own ship collided with another ship at 3′55′′. You can see that the value is increasing. Subject C observed the ship involved in a collision and maintained focus on the ship at 2′30′′. The other ship collided with the own ship at 3′30′′. The own ship navigated by subject C collided with another ship at 3′25′′. In 3′00′′ near 3′25, the value of the alpha-amylase concentration is lower; here, subject C did not have a risk perception of a collision with the other ship.

Discussion

The salivary alpha-amylase concentration of subject A at 3′00′′ indicates higher values in Fig. 8, compared to the preceding actual collision point with the other ship at 3′30′′. On the other hand, that of subject B at 3′00′′ also indicates higher values in Fig. 8, compared to the following actual collision point at 3′55′′.

Subject A recognized the risk collision at 1′30′′ and 2′50′′ in Fig. 8 and subject B did not recognise it at 0′40′′ in Fig. 8, the own ship navigated by both actually collided, however, salivary alpha-amylase did not respond to that and the concentration value before/after the collision. Therefore, we consider that alpha-amylase is a physiological index that does not respond unless there is an urgent situation under navigation, and that it would not be suitable as an evaluation index for the potential risk perception by human gaze movement itself.

Derived from this concept, an examination of subject C reveals that the alpha-amylase concentration at 2′00′′, reflecting the scenario where subject C anticipated a collision, exhibited a higher value than that observed at the actual collision time of 3′30′′. This observation suggests that alpha-amylase concentration increases in situations of urgency compared to the tension experienced during an actual collision. This phenomenon may reveal the lower value observed for Subject C at 3′00′′, which is the time point just before the maximum risk of collision at 3:25.

Consequently, human gaze movement is used this time before falling into a situation where there is a potential danger of a collision, such as the concentration of alpha-amylase increasing. As the subjects commented during the briefing, they were able to recognize other ships that may be in danger of colliding with them through their human gaze movement.

Conclusion

We confirm that alpha-amylase clearly responds to tension during collisions with other ships, but does not respond to collision their risk perception when recognizing other ships. In this respect, human gaze movement can be said to be suitable for risk cognition through subjects’ debriefing. The debriefing of the subjects for this experiment revealed that they clearly understood the risks through the screen. Previous research by Seta et al. has already shown that navigators have recognized risks in the actual ship maneuvering environment, so in future work, we will consider simultaneous human gaze movement measurement from the actual bridge navigation and remote ship maneuvering environment.

In this experiment, the risk cognition of collisions with other ships was successful because students becoming seafarers cooperated in the experiment, but we will use the same human gaze movement that demonstrated the effectiveness of risk cognition and verified whether amateurs can similarly recognize the risk through the screen. We believe that this will make it clear whether or not seafarers will need to be responsible for remote ship operations in the future.