Usability Evaluation for Drone Mission Planning in Virtual Reality

Abstract

There has been a lot of research conducted on human-robot interaction (HRI) with drones as well as human-computer interaction (HCI) with virtual reality (VR). However, little work was done on VR as HRI, even less for VR and drones in particular. In this paper, we evaluate the usability of VR interfaces used to control drones through working with UC Berkeley’s Immersive Semi-Autonomous Aerial Command System (ISAACS) project, which experiments with new ways for humans to interact with drones in a VR environment. Our experiment setting focuses on the drone mission planning phase and on creating an onboarding experience for new users. We develop a usability evaluation framework for the ISAACS VR system, and use this framework to conduct two iterations of user testing and prototyping with a human-centered design process.

The first three authors contributed equally to this work.

1 Introduction

Although VR is not a new technology, recent industrial buy-in and consumer uptake finally makes VR-based applications practical and affordable. As its hardware becomes more affordable and ergonomic, the VR market likely will continue to grow rapidly. Because VR places the user in a highly immersive environment, understanding its usability as an interface is crucial in the success of developing good VR experience.

Yet, many user experience professionals who are mostly familiar with designing for 2D interfaces often find that VR presents a challenging front. Designers and researchers have to consider how users perceive the virtual world, how to enable intuitive interactions, prevent motion sickness, and design and iterate on a VR application. With all the attention given to VR, there is currently still a lack of a standard usability standards when evaluating VR interfaces.

We believe that good VR design must start with a good understanding of both technology as well as human perception. Our current study to advance this understanding focuses on the drone mission-planning phase, as that is a key use case for an intuitive robot command and control system. In particular, we focus on building an onboarding experience for new users who would need more guidance.

2 Literature Review

HCI researchers have proposed several frameworks separately on human robotics interaction with drones and human computer interaction in VR. Germani et al. [4] proposed a schema that included four HCI main actors (tasks, human, equipment, and external worlds), where their metrics were divided into usability metrics and presence metrics. Murtza et al. [5] came up with 9 virtual reality heuristics, including synchronous body movements, physical space constraints, immersion, glitchiness, switch between real and virtual world, cord design, headset comfort, mental comfort, and user interface design. Norman’s classic framework of the seven stages of action [3] evaluated the usability based on the goal formation, three execution stages, and three evaluation stages. Since we are particularly interested in understanding how people perceive information and execute goal-oriented tasks in a step-by-step on-boarding experience, we adopt Norman’s classic seven stages of action to assess the perception and execution level of each task.

3 Metric Framework

Below is the metric framework that we develop based on Norman’s seven stages of action model (Table 1).

Table 1. Usability framework

The usability questions we investigate are:

• How easily can users tell what actions are possible?

• How easily can users perform specific tasks?

• How easily can users tell if the system is in a desired state?

We adopt Norman’s four design principles [3] for improving the VR interface:

• 1. visibility: a user can tell the state of the system and alternatives for action;

• 2. good conceptual model: there is consistency in presentation of operations and results;

• 3. good mapping: there is a clear relationship between actions and results, and between controls and effects; and

• 4. feedback: the system provides full and continuous feedback on the results of users’ actions.

4 Usability Evaluation

Based on our framework, we further develop our usability evaluation metrics. Table 2 is a summary of the heuristic evaluation.

Table 2. Evaluation metrics

5 Usability Testing Process

Each usability test takes about 30–40 min per person, where we start with learning about a user’s background experience in VR prior to the testing. During the test, we have the participants verbali their thoughts as they move through the user interface. This is a simple but effective method to discover what the user is thinking during the interaction and if there are any usability problems or confusion. At the same time, we also observe their performance and record notes and scores based on their performance. After testing, we ask the user to fill out two surveys. The first survey asks the user about their age, background in VR and drone flying, and how they evaluate their performance in the testing session. The second survey conducts the heuristic evaluation by comparing the interface against our defined usability principles (see Table 2). Based on the heuristic evaluation we conduct internally, we choose the questions that applied in this context and gather feedback on immersion, navigation, interaction, comfort, intuition, consistency, and clarity. We also record all testers’ virtual and physical interactions so we could compare and analyze their performance. All participants will be asked to complete the tasks listed below:

User Tasks

• 1. Set up a drone.

• 2. Set the first waypoint.

• 3. Adjust the height.

• 4. Modify the waypoint.

• 5. Add a waypoint on an existing route (“middle waypoint”).

• 6. Complete a flight route.

• 7. Remove a drone.

After the usability test, we then interview the testers for qualitative feedback on what went well and what did not.

6 First Round Usability Testing

For the first round of user testing, we test the original prototype which does not come with any onboarding tutorial to guide the user. The user is instructed to follow the step-by-step instruction from the facilitator. There are also no controller tooltips on the Oculus control to indicate how each button functions. In the starting steps, the user has to click on the joystick to retrieve the laser in order to begin tasks (Fig. 1).

Fig. 1.

First version that we user tested with.

6.1 Demographic

In this round, we have invited six participants between 19–29 years old with various background in VR and drone flying experience (Table 3).

Table 3. Demographic data from 1st round user testing.

6.2 Problems Identified

From the user testing, we define 4 simulated problems.

Add Waypoints

Without the controller tooltip, it was confusing for the user to start this task. In our design, the user need to start with adding a drone and then they would need to hover over the drone and click on button B to see the waypoint-adding option. It was not very intuitive. Most users thought they could click anywhere to start adding a new waypoint.

Adjust Height

After adding a waypoint, the waypoint would still be floating. The user can move their hands up and down to adjust the height before clicking on the button B again to finalize selecting the location. However, many users sometimes may double click and accidentally set the waypoint right after it was added. Due to the latency, the user would not notice they could make changes and moved on to the next task (Fig. 2).

Fig. 2.

Not enough visual feedback on adjusting the height.

Modify Waypoints

In the system, there are no visual cues to indicate how to modify waypoints. Prior to modifying waypoints, the users have to rely on the laser beam to select and click on objects for intended actions. However, they will not be informed that they could modify waypoints by holding and dragging them.

In addition, once the user accidentally set the waypoints to the ground, it could be difficult to modify them. Sometimes, these waypoints are set on the tables or outside of the area that the user could physically approach and modify (Fig. 3).

Fig. 3.

Once the waypoint was set on the ground, user sometimes could not physically reach to modify.

Add Mid-waypoints

To add a mid-waypoint, the user needs to click on the route in between two waypoints, click on button B again to add. The visual feedback on whether they are interacting with the waypoint or the route is unclear to indicate this interaction, which would lead to confusion.

Finally, in this version, there is no opportunity to remove waypoints. When the user sometimes tries to explore this function, they may accidentally add waypoints. Not being able to remove the waypoints is frustrating among all users (Fig. 4).

Fig. 4.

There’s no remove waypoint option in the menu.

7 Second Round Usability Testing

After revising the system based on the feedback from the first round of user testing, we conduct a new round of testing.

7.1 Demographic

In the second round, we invite six participants aged from 24–34 years old with diverse backgrounds in VR and drone flying (Table 4).

Table 4. Demographic data from 2nd round user testing.

7.2 Changes

We make four main changes for this round of testing.

Adding Tooltips

We add virtual tooltips anchored on the Oculus controller. The user could look at what interaction each button is associated with (Fig. 5).

Fig. 5.

Adding tooltips on the controller to indicate what interactions the buttons are associated with

Onboarding Tutorial

We also add a step-by-step tutorial so the user could follow the instruction on the board. Once they complete a task, the board will display the next step of the task. The purpose is to guide them through the planning task and be more familiar with the controls (Figs. 6 and 7).

Fig. 6.

Adding onboarding step by step instruction.

Fig. 7.

Following the instructions user will complete the tutorial with a complete mission planning.

Minimizing Control

In the first version, we used 5 buttons on the controller (Fig. 8). In this version, we remove the use of button B to decrease the memory load and learning curve of the user (Table 5).

Fig. 8.

Oculus controller buttons that are used in our VR prototype.

Table 5. Controls for version 1 and version 2

Visual Cues

We used 4 colors on the laser beam in the first version and the color mapping was not very clear to the user. In this version, we remove yellow in the system to have a more intuitive use of color as visual cue (Table 6).

Table 6. Visual cues on version 1 and version 2.

8 Results

To evaluate the usability of the system, we take a holistic approach comprising of: (1) data collected during testing, (2) a post-interview questionnaire, and (3) heuristic evaluation form for our internal team as well as a shorter version for users.

User Metrics

Table 7 shows the results collected from the user tests, based on these 7 user tasks. We select the following metrics to evaluate for each task:

Table 7. Overall test results

Completion Score:

The completion score (0–4) is a measurement of how successful the user was at accomplishing the task, with or without assistance. 0 indicated that the user failed the task. 1 indicated that the user partially completed the task with facilitator’s assistance. 2 indicated that the user partially completed the task without assistance. 3 indicated that the user completed the task intended with assistance. 4 indicated that the user completed the task intended without assistance.

Number of Errors:

Errors can be either slips (“fat finger”), mistakes, or user interface problems. Errors can occur more than once per user per task. We counted the number of errors occurred, even if it was the same error occurring multiple times, because that is an indication that the system lacks adequate feedback for user errors.

Error Rate:

The Error Rate calculation takes the total number of errors divided by the task complexity, which we’ve defined as the number of steps it takes to accomplish a task (e.g. number of clicks using the hand controller). The calculation for Error Rate is intended to convert errors into a proportion based on the opportunity for errors, with the idea that some tasks, especially those that are longer or more complex, will have more opportunities for users to make mistakes. This is partly borrowed from the concept used for THERP (Technique for Human Error Rate Prediction).

Time to Completion:

The time it took for a user to complete a task (either fully or partially).

9 Survey Results

Figure 9 shows the results from the post-interview questionnaire provided to users to capture subjective experience and perception.

Fig. 9.

Overall survey results.

Insights from User Metrics and Survey

Error rates have decreased for 70% of the tasks and Time to completion is shortened by 31 s for completing all tasks. However, we did receive mixed results for the Completion Score. After some reflection, we should conclude that this was likely due to a mix of factors: a small sample size of 12 participants with varying previous experiences in VR. In addition, since we provided an onboarding tutorial in VR during round 2, less instructions were given by the testing team, which suggests that the tutorial could be more explicit.

We found that error rates have improved for 5 out of the 7 tasks when an interactive onboarding tutorial were added, and participants reported lower stress levels and effort in round 2. This indicated that the onboarding tutorials helped reduce errors in almost all of the tasks. However, we did not find significant improvement in completion score, and the participants’ self-reported physical and mental activity were higher in round 2. This suggests that the onboarding tutorial could be more explicit - it is not sufficient for a tutorial to just tell users what to do. It is equally important to show them how to do it.

Comparing Test Results Based on Prior VR Experience

To dive deeper into the metrics, we split the results above and compared the results between participants with and without VR experience.

Table 8 shows the results comparing round 1 and round 2 for participants with prior VR experience, while Table 9 shows the results for participants without prior VR experience.

Table 8. Test results with VR experience

Table 9. Test results with no VR experience

For participants with prior VR experience, the results showed a decrease in time taken per task and error rates for 6 out of 7 tasks, which could be attributed to the consistency in visual cues and decrease in control inputs. However, we also found a decrease in completion scores for 5 tasks, which is likely due to the lack of explicit instructions in the onboarding tutorial.

For participants without prior VR experience, we found that error rates and time taken per task increased from round 1 to round 2. Compared to the results above, it is unsurprising that participants without any prior VR background are less equipped to complete the tasks on their own without a tutorial to show them how to do it.

10 Heuristic Evaluation

We conduct heuristic evaluation on our VR prototype internally. We also ask participants to self-report their overall experience in our prototype using a smaller subset of questions.

Internal

Figure 10 below shows the data collected when our team conducted a heuristic evaluation comparing rounds 1 and 2. Our internal heuristic evaluation revealed improvements in navigation, consistency, and intuitiveness. This is in-line with our initial hypothesis that adding an on-boarding tutorial would improve navigation, and minimizing control inputs would improve consistency and intuitiveness (Table 10).

Fig. 10.

Internal heuristic evaluation

Table 10. Minimizing control process in the system.

User Testing

Figure 11 below shows the data collected from user testing with a subset of heuristics. Users reported better results for intuitiveness and navigation, and less positive results for comfort and interaction. While we only tested this on a small sample size of users, the data is still in line with what we found during the user testing phase, where users reported inconsistent interactions when changing a waypoint versus adding a middle waypoint.

Fig. 11.

Heuristic evaluation from users.

Final Version

After the second round of user testing, we added some enhancements for the final version: opting to turn on the laser team pointer by default, and making the map bigger. Figure 12 shows the final heuristic evaluations chart done by our internal team, which revealed improvements in navigation, consistency, and intuitiveness. This is in-line with our initial hypothesis that adding an on-boarding tutorial would improve navigation, and minimizing control inputs would improve consistency and intuitiveness.

Fig. 12.

Heuristic evaluation comparison.

11 Usability Testing Insights

From two rounds of user testing, we synthesized our research into four key findings:

First, a step-by-step tutorial which walks users through an entire mission planning process is crucial in helping new users performed better. The tutorial helped to decrease the error rate and the completion time in 70% of the assigned tasks.

Second, as situational awareness is important during flights, minimizing input controls would help to decrease the error rate as it prevents cognitive overload. By using the same control button for a similar function instead of two control buttons, it helps reduce the learning curve for new users and makes it easier for them to remember the controls while experimenting with a new interface. (See Table 8 which shows the final version after two rounds of prototyping.)

Third, mission planning tools in VR should also have more interactions that imitate the real world, such as grabbing and dragging. Participants reported improved satisfaction, immersion, and intuitiveness when they discovered that that they could physically reach out, grab a waypoint, and then drag it to their desired location. Having interactions that is intuitive would be helpful in actual drone-flying situations.

Lastly, it is important for the VR system to provide immediate feedback and visibility for any system state changes during mission planning, as this would minimize any guessing on how the system works, which always leads to frustration and could potentially cause errors during the flight. In our example, during the second round of testing, the time to completion for adding a new waypoint increased from the first round. Even though we provided instructions to teach them what to do, we likely were not specific enough in our instructions. The users then started trying to guess the state of the system and how it works, which took them longer to complete the task and generally caused more frustration.

12 Future Work and Conclusion

Both usability testing and heuristic evaluation show that we could reduce users’ confusion and perceived ambiguity in our VR prototype by further improving the clarity of users’ status and providing more user controls in future iterations. First, based on the feedback received during our user testing sessions, we would like to enable users to remove waypoints, and build a consistent interaction between changing waypoints and creating a middle waypoint. Second, we plan to allow users to save a particular mission, and to record the process for future playback. From our initial research of existing 2D applications for drones, we would also add a pre-takeoff checklist and display safety concerns, such as height, incompatibility alerts, safety checks, and warning the user if the mission would exceed battery time. Finally, from our review of existing VR usability literature, we believe that adding audio and haptic feedback would provide a more immersive experience for the users. We hope to continue working with the ISAACS Human Interfaces team to conduct user studies on a larger scale to discover new ways of improving humans’ interactions with drones in a VR environment.

In addition to feature improvement, we believe that there is potential application for drone planning in VR.

• 1. Drone videography: we believe that there is a need for drone videography planning in a VR environment. From our interviews, planning for drone videography is currently either done via 2D apps, such as on the iPad, or using more basic techniques such as eyeing the landscape or looking over Google Earth. VR would provide a more intuitive way of planning their mission, as well as more control over the specific route, allowing videographers to preview shots and prevent accidents that would exceed safety height limitation.

• 2. Agriculture: farmers and agriculture professionals could use the system to monitor crop health or assess drought conditions.

• 3. Disaster management: disaster management experts could use the system to create a flight path that monitors cleanup progress and evaluates potentially dangerous areas.

• 4. Real estate and construction: Real estate and construction professionals could use the system to survey a property or monitor progress on a new development project.

• 5. 3D mapping: companies such as drone deploy are already using drones for 3D environmental mapping to create 3D models and high-resolution maps. The system could increase the level of immersion and intuition by allowing users to plan and save their missions in VR.

As the gap between virtual and physical worlds shrink in the use of VR with robotic systems like drones, usability become more complicated to evaluate. This paper provides a framework to address usability needs for a specific action sequence within the drone mission-planning phase, combining quantitative and qualitative analysis as well as heuristics to capture a comprehensive view of a subjective experience.