Thesis-Hector-VR/Concepts for operating ground based rescue robots using virtual reality/chapters/evaluate.tex

\chapter{Evaluation of User Experience}
\label{evaluate}

This chapter describes the design and detailed process of the user evaluation. The purpose of the performed user study is to measure the impact of four different operation modes on rescue efficiency, robot driving performance, and psychological and physiological stress and fatigue. For this purpose, participants are asked to find victims in a test scene using different operation modes and answer questionnaires after the test corresponding to each mode of operation.


\section{Study Design}

The evaluation for each operation mode consists of two main parts. The first part is the data recorded during the process of the participant driving the robot in the \gls{vr} environment to find the victims. The recorded data includes information about the robot's collision and the speed of driving etc. The rescue of the victims was also recorded. The second part is the questionnaires that participants filled out after each test. \gls{tlx} was used to measure the participant's subjective workload assessments. Additionally, participants were asked specific questions for each mode and were asked to select their favorite and least favorite operation mode. In order to reduce the influence of order effects on the test results, the Balanced Latin Square was used when arranging the test order for the four operation modes.



\section{Procedure}
\subsection{Demographics and Introduction }
Before the beginning of the actual testing process, participants were informed of the purpose of this work, the broad process and the content of data that would be collected. After filling in the basic demographics and signing a consent form, the features of each of the four modes of operation and their rough usage were introduced verbally with a display of the buttons on the motion controllers.



\subsection{Entering the world of VR}
After the essential introduction part, participants would put on the \gls{vr} headset and enter the \gls{vr} environment to complete the rest of the tutorial. Considering that participants might not have experience with \gls{vr} and that it would take time to learn how to operate the four different modes, the proposed system additionally sets up a practice pattern and places some models of victims in the practice scene. After entering the \gls{vr} world, participants first needed to familiarize themselves with the opening and closing menu, as well as using the motion controllers to try to teleport themselves, or raise themselves into mid-air. Finally, participants were asked to interact with the victim model through virtual hands. After this series of tutorials, participants were already generally familiar with the use of \gls{vr} and how to move around in the \gls{vr} world.



\subsection{Practice and evaluation of modes}
Given the different manipulation approaches for each mode and possible confusion between the different modes, participants would take turns practicing and directly evaluating each mode immediately afterward. 

The sequence of modes to be tested is predetermined. The order effect is an important factor affecting the test results. If the order of the operation modes to be tested would be the same for each participant, the psychological and physical exhaustion caused by the last operation mode would inevitably be more. In order to minimize the influence of the order effect on the results, the Balanced Latin Square with a size of four was used to arrange the test order of the four operation modes.

Participants automatically entered the practice scene corresponding to the relevant operation mode in the predefined order. After attempting to rescue 1-2 victims and when participants indicated that they were familiar enough with this operation mode, they would enter the test scene. In the test scene, participants had to save as many victims as possible in a given time limit. Participants were required to move the robot around the test scene to explore the post-disaster city and rescue victims. During this process, if the robot crashes with buildings or obstacles, besides the collision information being recorded as test data, participants would also receive sound and vibration feedback. The test would automatically end when time ran out or when all the victims on the scene have been rescued. Participants were required to complete the evaluation questionnaire and \gls{tlx} form at the end of each test. This process was repeated in each mode of operation. 

After all the tests were completed, participants were asked to compare the four operation modes and select the one they liked the most and the one they liked the least. In addition, participants could give their reasons for the choice and express their opinions as much as they wanted, such as suggestions for improvement or problems found during the operation.