The Do’s and Don’t’s of User Testing Moderation

For those of us who work in academia, it is a well known fact that over the summer part of your duties include supervising Masters students and supporting them in completing their design projects. And it is usually around the quiet(er) month of August that they are all at the stage of ‘testing’ their design concepts. What I have noticed over the years of helping first time user testing moderators, is that there is a considerable lack of practical information out there about how to be a good moderator on the actual day of the user testing session. There is a lot of resources out there about how to prepare your user tasks, how to write your scenarios, choose your metrics, analyse and present your data, but not enough practical info about how to actually run the user testing session.

I have been involved in a lot of user testing sessions over the years, both in academia and industry, and I have put together some basic advice about how to improve your moderating skills.

 

1.Do (over)prepare your introduction

A mistake I see often with first time or inexperienced moderators is that they neglect to understand the power of an in-depth introduction to the session. An inexperienced moderator might be more stressed on the day than the actual participant and they are generally too eager to ‘get on’ with session that they fail to brief participants about the set up and process of the session.  In order to get the best of your participants, it is important to make them feel relaxed and inform them of what they can expect to be doing, how long the session will take, the fact that the session will be recorded, what exactly they will be testing etc. Make sure you mention that they can be as honest and truthful as they can and that there are no wrong or right answers. I found in the past that a checklist of things to go through in the briefing session can sometimes be a lifesaver and helps with the first-time moderator nerves!

 

2.Don’t use leading questions

The most crucial aspect of obtaining valid user research is not to lead the user. Asking questions which prompt or encourage a certain answer will lead to false information. For example, asking ‘do you think this site is trustful?’ is asking for an opinion on something that the participant may not necessarily have been looking out for. Keep your questions open and allow the participant to speak their mind.

A classic example of a leading question that I have seen a lot of inexperienced moderators ask is :

Is it clear that this page/button/function is designed for …?

You might not realise but you if you ask such a question, you have already told the user the answer  YOU want.  When presented with a leading question, most users will be inclined to answer affirmatively.  Some users may say “no, it’s not clear,” but even, in that case, the data isn’t as good because you have limited the scope of the users’ response.  Most of all, you have missed out on the biggest opportunity in user testing – observing natural, undirected behaviour.  With a question like this, you have learned nothing about how a real user will interact with the site. So think twice about your questions, and if you have doubts about them, get someone else who is experienced in user testing to check your questions (and do not deviate from your questions protocol on the day of the testing!).

 

3.Do be aware of the Hawthorne effect!

A user testing session is artificial. no amount of realism in the scenarios, tasks, data, software or environment can change the fact that the whole thing is contrived. It is therefore important to understand the subtle (and sometimes not so subtle!) biases that creep into user testing. When users know they are being observed, they tend to exhibit slightly different behaviour than they would ‘normally’. This is known as the Hawthorne effect.

In the 1930s an experiment was carried out at the Hawthorne Works factory to explore whether light conditions increase the productivity of the factory workers. The experiments found that higher productivity was observed but this was not due to the changes in the lighting of the environment conditions. The conclusion was that the workers felt important because they were pleased to be singled out, and increased productivity as a result. Being singled out was the factor dictating increased productivity, not the changing lighting levels, or any of the other factors that they experimented upon.

So bear that in mind of the various biases that sometimes can affect the interpretation of your data. If this is something you re interested in exploring further, there are some pointers here: https://measuringu.com/ut-bias/

 

4. Don’t write notes in the middle of the session

This might be slightly controversial and I know a lot of experience user testing moderators that will disagree with me. In my opinion and from my experience, I think taking notes during a session that you are facilitating, will lead to participant bias. If a participant sees your write something down, they may think that their more recent action or comment was significant and will influence their behaviour by making them feel more self-conscious.

If a participant sees you write something down during a user test, they may think their most recent action was significant, influencing their behaviour by making them feel more self-conscious. Take a video recording instead and take notes from the video.

 

5. Don’t break the silence

 I am definitely guilty of this one, and this is where I personally went wrong for years when it came to my moderating skills. My fear of awkard silences resulted in me talking way too much, trying to fill every possible gap in the dialogue, including re-iterating and rephrasing what the participant just said, trying to finish the participant’s sentence when they struggled to articulate themselves and even explaining and sometimes ‘defending’ the design concept we were testing. As soon as I would start talking, I knew I was doing it but it would be too difficult to control myself and accept the silence. I can’t stress how important it is it to let the participant do the talking. The golden rule here is to shut up- allow the participant time and space to think, absorb, make judgement and generally think about what they are doing.

 

6. Do ‘boomerang’ technique to answer questions

As mentioned previously, it is really important to set the rules straight from the beginning of the session and let participants know that they can ask questions at any time (and take a break at any time), but that you won’t be able to help them with the tasks. Even if this sounds self explanatory, I can assure you that there have been numerous times where I got this seriously wrong, especially during user testing sessions with people with visual impairments. While trying to locate a function or a button on an interface while using a screen reader, participants would frequently ask me questions such as : ‘How do I go back to the menu?’ or ‘Where is the contact information’?

This is where an experienced moderator would ‘boomerang’ the question, and answer the participant’s question with another question, such as: ‘How do you think you would go back to the main menu?’ or ‘Where would you search for the contact information?’

 

7. Don’t defend the design!

I have to admit that I do see this a lot with our students who carry out user testing sessions for the design concepts that they have created themselves. After all that blood, sweat and tears they have shed during the design process,it is only natural to affiliate themselves with the product/system. They have after all, emotionally invested in it.

If this applies to you, when you find yourself trying to explain why the system/interface/product is not ‘performing’ as the user would expect it to, all the participant can hear is a ‘defence’ of the product. This prevents you from being a neutral observer and makes it more likely that participants will self-censor their feedback to not upset your feelings!

 If you really, really want to explain how to use the system or correct any misconceptions, then wait until the end of the session, once participants have tried it without your help.