Milestone 4: Evaluation

In order to evaluate our application, we conducted a heuristic evaluation, predictive evaluation, cognitive walkthroughs, retrospective testing interviews, think aloud evaluations, and issued a questionnaire to participants. We decided to test tasks specific to our application, such as recording a route, changing layers and reporting a hazard, because our target users are already familiar with some of the more general functions like searching and finding directions. The results of the tests are discussed below.

Heuristic Evaluation:

To conduct this evaluation, we had a member of our team go through the core tasks of the application and take note of issues encountered as they relate to the 10 heuristics we discussed in class. Our findings are discussed below.

Visibility of system status:
One problem noted is that when the record function is used, it does not indicate whether or not it is recording. Also, when you click on the menu, stop and pause are options even before the user clicks record. This could be confusing if the user does not remember whether or not they started recording their route.
Severity Rating: 4
Use of familiar metaphors and language:
The application uses very simple language. There is no computer jargon or unfamiliar terms that are used. The only thing that could be slightly more clear is that on the report menu, one of the options listed is an acronym. It says “POI” for point of interest. This could potentially be confusing if the user does not know the acronym, but the application also has a description of each option below, so the user should be able to discern the meaning. The layers option is somewhat ambiguous and may not initially make sense to users that are new to hiking, but will quickly become clear. Experienced hikers will understand the purpose.
Severity rating: 1
User control and freedom:
The applications controls are all easily accessible at the bottom of the screen. Most functionality does not do anything permanent so there is no need to ask the user to confirm before completing a task. When reporting something, there is a back button at the top of the screen which the user can click to cancel the report. Users can easily access the features of the app or quickly return to the homescreen.
Severity rating: 0
Consistency and standards:
The layout of the application is pretty consistent with applications that are commonly used by our users. Having a menu on the bottom of the screen is typical because the users fingers will be closer to it, so it is easily accessed. We used an API for the mapping that resembles Android maps, so it will be very familiar to Android users. Further, buttons like find location, maximize, and minimize are recognizable button images. The layout is consistent across each of the options in the application.
Severity rating: 0
Error Prevention:
In order to prevent errors, in the reporting forms the users are given choices for most of the fields rather than entering in whatever text they want. The user is allowed to expand upon their report, but for important classification items, they can select options from a menu. The only problem noted with this heuristic is similar to the one for visibility of system status. If the user opens the record menu and presses “pause” when the application is not recording anything, the application will not respond. While this does not technically cause any problems, it would be a user error, and the menu should change based on whether or not the application is recording.
Severity rating: 1
Recognition rather than recall:
With the record function, if the user forgets whether or not they started recording their route, that could be an issue. This function should be changed so that when the application is recording, it is indicated on the map, and so that the menu options do not show pause/stop if the user is not recording yet. Other than that, there is no instance in which the user would need to remember any information in order to use the application.
Severity rating: 4
Flexibility and efficiency of use:
The application does not have any shortcuts. Because most of the forms have predetermined options for users to choose from, that makes the application slightly less flexible, but this is remedied by the option to add more information. The application is very basic and all of the functionality provided is necessary, there are no extra, unimportant functions.
Severity rating: 0
Aesthetic and minimalist design:
The application has a very minimalist design. Aside from the colors on the map, the only colors used are black, white and green. This adds to the consistency of the application’s interface as well. There is also only one font used. A little variation in color and font could make the design more aesthetically pleasing, but overall it doesn’t look bad and the design make it clear how to complete certain tasks.
Severity rating: 1
Helping users recognize, diagnose and resolve errors:
There is not an instance in which the user can make an error with this application.

Severity rating: 0

Help and documentation:
The application does not have a help tab in it. On the reporting menu, there are small descriptions of each options that can help point users in the right direction. There is no documentation for the application either. The functions of most of the features are clearly stated to users.
Severity rating: 1

Predictive Evaluation:

To conduct this evaluation, we used the KSLM Model to predict how long it would take the user to complete two tasks: Reporting a hazard and changing the map layer. We used the times from the following table, and used the average for pressing a single key.

Reporting a hazard:

H Homing hand on the mouse
P Pointing to the report button
P1 Clicking mouse
P Pointing to the obstruction option
P1 Clicking mouse
P Scrolling below the picture to the form
P1 clicking on drop down menu for obstruction type
P Moving mouse to click on option
P1 clicking on option
P moving mouse to the difficulty field in the form
P1 clicking on drop down menu
P moving mouse to level of difficulty
P1 clicking on level of difficulty
P moving mouse to submit button
P1 clicking on submit button
Calculation: 1H + 7P + 7P1 = 0.40 + 7(1.10) + 7(0.20) = 0.40 + 7.70 + 1.40 = 9.50s
*Note: process length may vary if the user chooses to add the optional description. In that case, the time would depend on the length of the message.

Changing map layer:

H Homing hand on the mouse
P Pointing to the layers button
P1 clicking mouse
P pointing to desired layer
P1 clicking on mouse
Calculation: 1H + 2P + 2P1 = 0.40 + 2(1.10) + 2(0.20) = 0.40 + 2.20 + 0.40 = 3.00s

These are relatively short times to complete tasks, and we believe that our design has enabled efficiency for the user. Changing the map layer could probably not be any shorter, but after evaluating the reporting task, it is clear that it could be 1.10s faster if the user did not have to scroll to below the picture before they start filling out the form. This is something to consider if we were to revise our design.

Cognitive Walkthrough:

As a user, my goal here is to use the web app when I go hiking to track my trail and mark the places where I spot an incident.

Goal: Track my trail (record & navigation)

Open Active app
Click on Navigation button on bottom of screen
Type in start location in the “Direction From” slot
Type in end location in the “Direction To” slot
From there, the full trail you wish to hike will show up on the map.
Once you get to the start location and you want to record your trail, click the Record button at the bottom of the screen.
Click Start when you want to start your recording
Click Pause when you want to pause your recording
Click Stop when you want to stop your recording
Once you click stop, the recorded trail will be saved in the app

Goal: Mark places where I see an incident

Open Active app
Click the Report button on the bottom of the screen
From there, you will see the options: Obstruction, Detour, Trail Conditions, POI, Trail Review
Once you click on any of these options, there will be a form to fill out and it is very straight forward with fields telling you what information needs to be filled out.
Once the form is filled out, click the submit button
If the form submitted was for Obstruction, Detour, Trail Conditions or POI, then that information will be seen on the main map on the main page.
If the form submitted was for Trail Review, then that information will be seen when the user clicks on the trail and wants to see reviews.

Retrospective Testing Interview:

To conduct this evaluation, we had our team review the video recordings of a usability test with them present while asking the user to explain their behaviour during the test. The usability test consisted of two tasks: reporting a hazard and changing the map layer. The retrospective testing interview takes place in a quiet room with a TV, table, and notepad for note taking.

While watching the video recordings with the user, they seemed stressed. When asked about it, they mentioned that watching and hearing themselves on a video is strange to them. For reporting a hazard, the user takes a bit of time locating which report button to press. The user states that they were looking for the ‘hazard’ button and could not find it, so they clicked Obstruction using their best guess. Aside from that, the user mentions that it was easy to type in the kind of hazard with more information and was easily able to submit the obstruction with little hassle.

When presented with the video recording of changing the map layer task, the user takes a second to think about what a map layer is. The user informs us that they were unsure what the map layer was at first, but having it on the bottom menu bar helped them figure it out. Once they clicked the layer button, they understood what the function does and seemed to be less stressed.

Think Aloud Evaluation:

To conduct this evaluation, we had a member of our team ask participants to user the prototype while speaking their thoughts out loud. The evaluation takes place in a quiet room with a table, the prototype, and notepad for note taking. The task they were asked to complete involved tracking their path and reporting findings along the way. In order to be able to complete this task in a static indoor environment, the thing the user “encountered” along the trail was described by the team member. The user was then asked to pretend that they had found them. Our findings are discussed below.

User #1:

56 yr old female. The user thought finding the button to record a track was easy to find, since it matched the wording she would have used to describe the action. When it was clicked, she commented on the smooth animation. There was no hesitation when the user clicked on “Start” to start the recording. When an obstruction (described as a broken bridge) was “seen,” she immediately found the report button. She was confused by the next screen, where she had to choose a category for the report. She was deciding for several moments between an “obstruction,” “detour,” and “trail conditions.” She tried to ask the test operator, but was told to try to complete the activity herself. She then chose “obstruction,” where she was able to fill out the information about this particular incident easily. Stopping the trail recording was also seamless, as she saw the stop option earlier.

User #2:

26 yr old female. This user initially tried to click “layers” when trying to find the option to record a trail before trying again on record. Since the record option is still visible in the layers menu, very little time was lost. When the broken bridge obstruction was described, the user just described what she was doing as she quickly clicked on the correct options to enter the details about the bridge. She was only confused if adding a picture was optional. Since we did not have a picture available, she quickly found out that it was.

User #3:

24 yr old male. The user had no problems at all using the system. He was able to find the options for starting and stopping the tracks immediately, and had no problems finding the menu and entering the information for the obstruction. He did not talk much during the experience since it did not take him long at all.

User #4:

20 yr old male. This user pretty quickly found the option for starting the tracks, but chose to look through most of the rest of the interface of the system to see what kind of options were available. Due to this, when he was instructed to enter the details about the bridge, he was quickly able to find the form. He was, however, slightly confused about the “How navigable” option, since he did not remember the exact description of the obstacle. After getting the information again, he still wasn’t sure which option to pick. He eventually picked “Not at all,” but wasn’t sure about his choice.

User #5:

21 yr old male. This user began with easily finding the record trail option. He commented that he liked seeing a “pause” option for things like lunch breaks or other rests. After the obstacle was described, he went straight to the report option. He was confused momentarily about which category to choose, but decided on “Obstruction.” He then was able to choose the type of obstruction, but had some difficulty deciding on the navigability, eventually deciding on “Not at all.” He commented that he didn’t know what the bridge really looked like, so he didn’t want to add comments in the description box. Stopping the trail recording was also straightforward for him.

Questionnaire:

For the questionnaire, we first wanted to get some demographic information so we could see how participants of different ages, genders and experience with computers struggled with the system and what they liked. Next we asked how easy the system was to use overall, and had them rate it from 1 to 10, with 10 being very easy. We asked each participant for something that they struggled with, and left this open ended as it could have been very different for each of them. Next we asked how successful the participant felt after completing the tasks, because user sentiment determines how likely the user is to reuse the app or to recommend the application to their friends. Finally, we had them rank the importance of each feature. In our initial planning stages, we determined which features were most important to potential users. By asking this at the end, we can use the results to evaluate whether or not we designed the application in such a way that these needs were met.

Results:

Overall, the design of this application was successful. Through our analysis, we identified a few minor details that we would change if we were to implement this prototype. The first was the recording error discovered in the heuristic evaluation. While in the cognitive walkthrough, users did not have issues with this, however it could be confusing if there is no feedback provided when a user begins and stops recording a route. If the user forgets whether or not they began recording, they could mess up the recording. This is a visibility problem and will be resolved. To fix this issue we would add a little box in the upper right corner with a red dot that says “recording” to indicate that the application is recording the route. The second issue is that there are too many categories when reporting something. 40% of the participants struggled while deciding which category to place the broken bridge obstruction in. This error rate is too high and in the future we would provide fewer, more general options to make this choice easier. Adding an option like bridge/path obstruction would help solve this problem. Further, an “other” section where the user can describe the problem along with the picture will help cover for more obscure obstructions. Something we did really well was making all choices visible and available to users. None of the participants had problems finding the options necessary to complete their tasks. If we were to implement this prototype, we would keep the general layout of the application the same. We like the minimalist design and quick, user-friendly interface.