Tuesday, August 22, 2017

User Testing: Our Team’s Process for Virtual and In-Person User Testing

Lillian Krieger, Lead Product Designer

User testing is something that we feel strongly about at Rightpoint. Both design and development teams conduct user tests to determine how target users will respond to and interact with a product. We’ve been designing and building products since 2003. User testing isn’t something that’s new to us, but we’re constantly trying to come up with new ways to organize and execute our process. A part of our process includes sharing how we work with both our clients and the design and development communities. Below is our example setup for in-person, moderated user testing.

Sourcing Users

There are many ways to source users for rounds of testing.

In this post, we’ll talk specifically about:


 What is it?

Userinterviews.com is a website that allows you to source users from your area for in-person or remote testing.

● Userinterviews charges $20 per sourced tester and you need to give an extra incentive pay to each user (gift cards work well!).

How it Works

1. Set-up screener questions and select times you are available. (leave time in between tests as a buffer in case you go over and to greet the next tester – at least 15 minutes. Also, request more testers than you need in case someone cancels.)
2. It goes live and people will go through the screening questions.
3. You will receive a list of screened possible testers, and then you can approve them.
4. Approved participants will be notified and select a time – their status will then become “CONFIRMED” on the dashboard.
5. Userinterviews will give you a schedule with contact information & demographics for each participant.

Benefits of Userinterviews

● Includes Screener Questions
● Vetted Participants
● Automatic Scheduling
 Users are charged after the interview is complete (no charge for no-shows)
● Sources users within a few days



 What is it?

Usertesting.com is a website used to conduct unmoderated, remote testing. They also have mobile technology (which we helped build here at Rightpoint). While the user goes through the test, it records audio and the screen input. This allows you to better see and understand the actions users are taking with your product.

Setup for In-Person Testing

screenshot of usertesting.com showing the UserTesting panel and My Recruit panel

1. Create a new test and select “My Recruit.” This will create a link that you can send to specific people.
2. When setting up the test for moderated testing, don’t worry about filling out the tasks, since the user will never see this. We wrote, “listen to moderator”. Questions will be asked by a moderator rather than UserTesting.
3. It will generate a link. Send this link to an email address that you can access on the phone that will be used for testing.
4. Before the interview, click the link in the email and login. You will need a different email for every test, so send as many beforehand as you’ll need.
5. Click through the initial onboarding of the test. Hit the “Start” button (recording will begin) and then pause the recording. Load InVision. – Leave like this until your user comes in.
6. When your user is ready hit the “Resume recording” button and record your initial background questions, even if they aren’t interacting with the app.
7. When the user test finishes, conclude the test on the phone, and you are done!


● All user testing information is cataloged in one place
● Shows touch points in the recording
● Records audio and screen

 Technical Setup

1. Create a prototype in InVision (or the design software of your choice).
2. Test prototype through Usertesting.com to record and save information.
3. If you want others to listen live:
Hook up the device to a laptop and use Quicktime to show the device’s screen on your laptop.
Setup Google Hangouts and screenshare the window to broadcast to outside listeners. (We recommend that you only have 1-2 people in the room with the user to ensure a clear recording. Everyone else can listen at their desk with headphones.)

Physical Setup

● Print the following:
○ Consent forms – Draw up and send these to users ahead of time so they can read through them! Having an email connection with the users before the test day is also a great way to start building a solid foundation for rapport.

○ Signs to make navigation the day of the test easier. We usually have these in the elevator banks, leading to testing rooms, to the restrooms, and most importantly on the doors of testing rooms so that testers aren’t disturbed.

○ Surveys – These will differ depending on the product being tested, but we typically ask users to strongly agree or strongly disagree with key statements about the product in general and about  ○ List of testers to give to the front desk – this is an important organizational element. Not only does it make the process of the users getting to the office easier (through the lobby security), but also helps to keep things on schedule throughout the day. Feel free to give blocks of time a bit of padding so your team can organize thoughts before the next user arrives.

● Materials

○ Laptop

○ Charger 

○ Phone (big screen preferred)

○ Lightning cord ( or whatever is needed to connect the phone to laptop)

○ Phone stand

○ Clipboard for surveys

○ Pens for surveys and consent forms


●  Buy

○ Payment for users (we use Amazon.com gift cards)

○ Water bottles for users

○ Snacks to grab


● Other:

○ Sticky notes, pens, & dry erase would be smart for outside listeners to start generating thoughts for affinity diagrams.

○ Business cards, in case any of the users have follow-up questions.


Things We’ve Learned From Doing This

● Send a personal email the day before testers come in. It was nice to have some responses and connections to the people coming in – and remember, be friendly!

● Send consent forms prior (as well as having them in-person) so you don’t waste time with them having to read it during your session. Testers also need a heads up so they aren’t asked to sign blindly.

● Schedule AT LEAST 15 minutes between users to allow for transition time, running over, etc. This is the padding time we mentioned earlier.

● Have a person other than the moderator be ready to greet and walk people out in case they go over time is helpful. The 2nd person can also give out a survey and do a closing talk with testers asthey leave.

● 33% of our users canceled, so schedule a few more testers than you need (more data is better than less!).


Make Sure you

● Have the moderator listen to all of the recordings to take notes 

● Have all listeners write quotes, findings, pain points on sticky notes

● All Listeners come together with sticky notes and create an affinity diagram (even if it’s just 1 listener!)

○ Group stickies with common themes. For example, we uncovered patterns in people’s thoughts about trust within a recent app we tested, as well as self-reflection.

● Talk about each sticky and groupings as a team. This creates good discussion and gets everyone on the same page.

● Create a write-up This is a helpful exercise when there is a lot of information, and lets you compare tests in a more visual and interactive way to find patterns. _____________________________________________________________________________________________________________________________

Do you have a project in mind? We’d love to work with you. If you’d like an opportunity to work on projects with us, check out our Careers page. We’re hiring!