Usability Testing:
ROM Guidance Panel

Usability Testing:
ROM Guidance Panel

Usability Testing:
ROM Guidance Panel

Capturing user behaviour and feedback to improve features of a beta mocap software before official release in Entertainment, VFX and Media markets

Capturing user behaviour and feedback to improve features of a beta mocap software before official release in Entertainment, VFX and Media markets

Project Scope

Project Scope

Timeframe: 6 weeks

My Role: Lead UX Researcher

Team: Ed (PM), Anita (QA), Yuga (PD), Aya (Marketing Manager)

Methods: Contextual inquiry, moderated in-person usability testing, qualitative analysis

Artefacts: Research session protocol, scripts, note-taking templates, consent form, findings report

Tools: Miro, Zoom, MS Office, Confluence

5

Customer visits

Customer visits

13

Users interviewed

Users interviewed

21

Questions answered

Questions answered

Project Overview

'How would users react to our solution?'

Product

The product in focus was Shōgun, the company's world-leading motion capture software designed for media, entertainment and VFX production applications. Our teams have continued to work on powerful functionalities that help users increase autonomy over volume operations and data analysis, to cut down on production time and costs.

The product in focus was Shōgun, the company's world-leading motion capture software designed for media, entertainment and VFX production applications. Our teams have continued to work on powerful functionalities that help users increase autonomy over volume operations and data analysis, to cut down on production time and costs.

Kickoff

Our development and algorithm teams had built a new feature integrated in Shōgun's upcoming version update: the ROM guidance panel, which visualises subject calibration information (‘range of motion’) in real time capture.


They wanted to know how end users would interpret, learn and use the feature.

Our development and algorithm teams had built a new feature integrated in Shōgun's upcoming version update: the ROM guidance panel, which visualises subject calibration information (‘range of motion’) in real time capture.

They wanted to know how end users would interpret, learn and use the feature.

Objectives

  1. Understand user behaviour, decisions and experiences in subject calibration processes


  2. Identify usability and UX problems in the ROM guidance panel prototype


  3. Uncover opportunities to improve Shōgun design

  1. Understand user behaviour, decisions and experiences in subject calibration processes


  2. Identify usability and UX problems in the ROM guidance panel prototype


  3. Uncover opportunities to improve Shōgun design

Notes

My discussions with colleagues revealed that no user research had been done to understand target users, and that no design artefacts had ever been created at the company.


Because of this, part of my work focused on collecting data to create evolvable personas and to document use case scenarios.

My discussions with colleagues revealed that no user research had been done to understand target users, and that no design artefacts had ever been created at the company.


Because of this, part of my work focused on collecting data to create evolvable personas and to document use case scenarios.

My discussions with colleagues revealed that no user research had been done, and that no design artefacts had ever been created at the company.

Because of this, part of my work focused on collecting data to create evolvable personas and to document use case scenarios.

Methodology

Methodology

  • Stakeholder interviews

  • Contextual inquiry + moderated in-person usability testing

  • Data analysis

  • Report writing & presentation

To learn how Shogun worked, I asked to go through a number of key journeys myself under a mocap technician's instructions - like some target user types do. Here is a cropped, sped-up video of me performing subject calibration in a volume (the costume is not embarrassing at all).

Stakeholder interviews

As the company's new and first UX Researcher, I made it a priority to understand the project's context, history and goals. I also wanted to outline communication preferences and align on a shared vision for success before jumping into research planning.

I informally approached our Senior Engineer, Lead Developer, Product Manager, and Head of Software for entertainment products, and conducted informal semi-structured 1-1 interviews following a discussion guide to ensure deep and free-flowing conversations centred on four major topics:

  • Project history and priorities to address in Shōgun

  • ROM guidance panel

  • Success metrics

  • Expertise, process and workflow


With the small number, it was rather quick and easy to outline major themes via lightweight analysis. High-level insights from these internal interviews were organised as part of the key Shogun UX research document and shared in the depository for larger teams to access and evolve in time. This instigated a zeal for documentation of informal discussions across the Devs and Product teams.

Stakeholder interviews

As the company's new and first UX Researcher, I made it a priority to understand the project's context, history and goals. I also wanted to outline communication preferences and align on a shared vision for success before jumping into research planning.

I informally approached our Senior Engineer, Lead Developer, Product Manager, and Head of Software for entertainment products, and conducted informal semi-structured 1-1 interviews following a discussion guide to ensure deep and free-flowing conversations centred on four major topics:

  • Project history and priorities to address in Shōgun

  • ROM guidance panel

  • Success metrics

  • Expertise, process and workflow


With the small number, it was rather quick and easy to outline major themes via lightweight analysis. High-level insights from these internal interviews were organised as part of the key Shogun UX research document and shared in the depository for larger teams to access and evolve in time. This instigated a zeal for documentation of informal discussions across the Devs and Product teams.

Stakeholder interviews

As the company's new and first UX Researcher, I made it a priority to understand the project's context, history and goals. I also wanted to outline communication preferences and align on a shared vision for success before jumping into research planning.

I informally approached our Senior Engineer, Lead Developer, Product Manager, and Head of Software for entertainment products, and conducted informal semi-structured 1-1 interviews following a discussion guide to ensure deep and free-flowing conversations centred on four major topics:

  • Project history and priorities to address in Shōgun

  • ROM guidance panel

  • Success metrics

  • Expertise, process and workflow


With the small number, it was rather quick and easy to outline major themes via lightweight analysis. High-level insights from these internal interviews were organised as part of the key Shogun UX research document and shared in the depository for larger teams to access and evolve in time. This instigated a zeal for documentation of informal discussions across the Devs and Product teams.

Contextual inquiry + usability testing

Contextual inquiry + usability testing

Gathering all internal stakeholders' questions about the ROM Guidance Panel (hereby ‘RGP’), I rephrased them to form research questions ('RQs'). The RQs were synthesised under two main categories, respectively pertaining to general UX and product-specific reaction. This called for a combination of methods that would allow us to effectively uncover issues and opportunities at both levels of granularity.

GOAL 1

Unveil the human, environmental, and contextual factors affecting subject calibration processes, results, and user experiences

GOAL 2

Capture behaviour and interactions in context with metrics to assess RGP impacts, to uncover issues and areas for improvement

To learn how Shogun worked, I asked to go through a number of key journeys myself under a mocap technician's instructions - like some target user types do. Here is a cropped, sped-up video of me performing subject calibration in a volume (the costume is not embarrassing at all).

To learn how Shogun worked, I asked to go through a number of key journeys myself under a mocap technician's instructions - like some target user types do. Here is a cropped, sped-up video of me performing subject calibration in a volume (the costume is not embarrassing at all).

To answer the RQs, I considered field research necessary and hence worked with our entertainment PM to arrange a series of visits. Given the rare opportunities we had to see customers on site, I decided to combine contextual inquiry and usability testing to maximise findings in a single session.

I created the protocol, script with a comprehensive list of tasks and follow-up questions, note-taking template, metrics guideline, client-facing presentation and consent form to share in advance. The 1.5-hour session was outlined in three parts:

Read session script


PART 1: FIELD OBSERVATION & INQUIRY

PART 1, PREPARATION

1.1 Observation
 - Volume setup: Hardware & software
 - Camera warmup
 - Actor(s) suit-up

 1.2 Inquiry
 - Shooting process and workflow (focus on setup)
 - Environment, system, people, event and schedule factors

1.1 Observation: Volume setup (hardware & software), camera warmup, actor(s) suit-up

 1.2 Inquiry: Shooting process and workflow, environment, system, people and event factors


PART 2: USABILITY TESTING

PART 2, USABILITY TESTING

2.1 Task one
 - Subject calibration with 1.10 + follow-up Qs
2.2 Task two
 - Subject calibration with RGP prototype without demo + follow-up Qs
2.3 Task three
 - RGP demo + subject calibration with RGP + follow-up Qs

2.1 Task one: Subject calibration with 1.10 + follow-up questions

 2.2 Task two: Subject calibration with RGP prototype, no explanation + follow-up questions

 2.3 Task three: RGP demo + subject calibration with RGP + follow-up questions


PART 3: DISCUSSION

PART 3, DISCUSSION

3.1 RGP
- General experience with RGP and feedback

 3.2 Edge cases
 - Use scenarios & potential challenges (e.g. subjects of limited ROM, custom marker sets)

 3.3 Wrap-up
 - Questions and future visits

3.1 RGP: General experience with RGP and feedback

 3.2 Edge cases: Scenarios and potential challenges (e.g. limited ROM, customised marker sets)

 3.3 Wrap-up: Questions and future visits

Data analysis

Immediately after each session, I debriefed with our team to organise rich summaries of digital, paper notes, audio/video files, and surface preliminary codes. The recordings were transcribed via Otter.ai and manually edited; all data were collated, formatted for consistency, and imported to NVivo for systematic viewing and processing. Metrics were gathered in one Excel file.

First running an analysis of the metrics (time on task, errors, SUS), I then conducted thematic analysis over a week, coding user statements as well as observer descriptions. I started with our predefined codes of larger categories ('subject calibration context' and 'RGP') which facilitated theme identification while leaving room to explore data and challenge preconceived notions.

Next, I reviewed the extracts of coded data under devised themes to reform or refine them in terms of validity, coherence, clarity and relevancy—that is, whether each had been effectively derived data, appropriately grouped together, adequately descriptive of our understanding, and possessive of the explanatory power to answer our RQs.

Finally, I wanted to reduce my personal biases in interpreting the data. I involved colleagues across Product and UX to critique the themes with fresh eyes over one week, providing a main guideline:

  1. Is the theme well supported by the data (saturated with lots of instances)?

  1. Are the themes well distinct from each other, and relevant to building Shōgun?

  1. Looking at the data, do you agree with the themes? Any missing picture?

Data analysis

Immediately after each session, I debriefed with our team to organise rich summaries of digital, paper notes, audio/video files, and surface preliminary codes. The recordings were transcribed via Otter.ai and manually edited; all data were collated, formatted for consistency, and imported to NVivo for systematic viewing and processing. Metrics were gathered in one Excel file.

First running an analysis of the metrics (time on task, errors, SUS), I then conducted thematic analysis over a week, coding user statements as well as observer descriptions. I started with our predefined codes of larger categories ('subject calibration context' and 'RGP') which facilitated theme identification while leaving room to explore data and challenge preconceived notions.

Next, I reviewed the extracts of coded data under devised themes to reform or refine them in terms of validity, coherence, clarity and relevancy—that is, whether each had been effectively derived data, appropriately grouped together, adequately descriptive of our understanding, and possessive of the explanatory power to answer our RQs.

Finally, I wanted to reduce my personal biases in interpreting the data. I involved colleagues across Product and UX to critique the themes with fresh eyes over one week, providing a main guideline:

  1. Is the theme well supported by the data (saturated with lots of instances)?

  1. Are the themes well distinct from each other, and relevant to building Shōgun?

  1. Looking at the data, do you agree with the themes? Any missing picture?

Insights

Insights

Insights

I fleshed out 9 major themes, each comprising 3-5 subgroups, to answer our RQs under the three research objectives:

  1. Calibration journeys and contexts

  2. RGP usability and UX

  3. Opportunities for Shōgun


My findings informed key iteration decisions surrounding the v1.11 beta prototype, helping the Product and Dev teams to evaluate their backlog and prioritise tasks based on evidence and design impacts.

User pointing to an RGP issue

User pointing to an RGP issue

User reaction to RGP

  • Satisfaction

  • Issues & pain points

    • UI & interaction

    • IA & labels

RGP in edge cases

  • Limited ROM

  • Custom marker sets

  • Blockers & separate viewports

RGP in edge cases

  • Limited ROM

  • Custom marker sets

  • Blockers & separate viewports

Calibration UX factors

  • Workflow

  • Setup

  • Distractions

  • Shooting considerations

System / technical issues

  • Floor contact

  • Spinal alignment

  • 3D workspace

  • Auto skeleton labeling

System / technical issues

  • Floor contact

  • Spinal alignment

  • 3D workspace

  • Auto skeleton labeling

Learning behaviour & touchpoints

  • Novice

  • Expert

  • Non-technician

Support requirement

  • Advanced UI

  • Training materials (staff)

  • Education materials (clients)

Support requirement

  • Advanced UI

  • Training materials (staff)

  • Education materials (clients)

Example themes (descriptions omitted)

Report & Presentation

For NDA reasons, I am unable to share the final report and presentation, or specific metrics, product insights, design suggestions and artefacts created in this project. Click on the following to see (1) an early report of synthesised data from a single customer visit, or (2) an example user journey that I would create to keep stakeholders informed of initial findings.

Read a customer visit summary

Explore other projects

Generative Research: Mocap System

Generative Research: Mocap System

Analysing how users in the life sciences community adopt motion capture technology, employing qualitative methods—assumption mapping, field observations, inquiries, and workshops—to create artefacts that ground teams with data in a product redesign.

Analysing how users in the life sciences community adopt motion capture technology, employing qualitative methods—assumption mapping, field observations, inquiries, and workshops—to create artefacts that ground teams with data in a product redesign.

Read about Nexus research

Discovery: Uncovering Insights

Discovery: Uncovering Insights

Delving into the depths of user experiences in China's knowledge entertainment market to discover behaviour and desires, identify pain points, explore market gaps, and inspire an sustainable offering that fits both user and business needs.

Delving into the depths of user experiences in China's knowledge entertainment market to discover behaviour and desires, identify pain points, explore market gaps, and inspire an sustainable offering that fits both user and business needs.

Read about Know+ discovery

HCI: From Theory to Application

HCI: From Theory to Application

Applying human-computer interaction theories in projects to showcase my knowledge and skills in user research, system evaluation and audit, inclusive design, interaction design, IA, and web app development.

Applying human-computer interaction theories in projects to showcase my knowledge and skills in user research, system evaluation and audit, inclusive design, interaction design, IA, and web app development.

Read about academic projects