Old Wine in New Bottles or Novel Challenges? A Critical Analysis of Empirical Studies of User Experience

Author

Javier A. Bargas-Avila, Kasper Hornbæk, CHI 2011

Keywords

User experience, UX, Usability, HCI, Review, Meta-Analysis

WHAT

overview of the current state of UX research

WHY

to explore the current state of UX studies about methodology and content
to clarify what UX is, survey approaches

HOW

QUOROM for meta-analysis
- Phase 1 - Potentially relevant publications identified
- Phase 2 - Publications retrieved for detailed evaluation
- Phase 3 - Potentially appropriate publications
- Phase 4 - Publications to be included in the analysis

Result

what sections are included in the paper about UX
- Products, use situations and context in UX research
  - Product Types
  - Use Situations
  - Context
- Dimensions of Experience in UX research
  - Dimensions
  - Number of dimensions involved
  - Usability metrics
- How are data on UX collected?
  - Qualitative (50%), Quantitative (33%), Mixed (17%)
    - Methodology changes by time
  - Data-collection Methods
- When is UX assessed?
  - before, during, after and imagined
  - only 17% on all situations
  - Anticipated use
  - No temporal studies done over periods of time
- Main Goals
  - Basic Research
  - Methods
  - Expectations and requirements
    - Not considered as a dimension of UX

Discussion

Products, use situations and goals
- A heavy emphasis on art and consumer products
- The turn to art and consumer products may preclude conclusions about experience with other types of products
- The contribution to the understanding of a broad amplitude of products is largely ignored
- Non-instrumental, hedonic or non-task-oriented goals are associated with UX
  - instrumental, pragmatic or task-oriented goals are associated with usability
Dimensions of experience
- Emotion is one of the key factors of UX
- Aesthetics and hedonic quality are strongly overlapping constructs
- Dimensionality explosion
Methodology
- Uniqueness studies and dimensions studies
- Overemphasize methodological stance to the extent of damaging research quality
  - Uniqueness papers
  - Dimensions papers
  - Few studies combine the methods of uniqueness and dimension studies
- Suggestion
  - The proposals for new methods are rarely validated
  - Constructive methods are not only rarely validated, they raise many issues about the process and validity of interpreting their results.
  - The need to rely on first-person methods for understanding experience
Limitations and future work
- Limitations
  - Ignore how practitioners work with UX
  - The discussion of individual studies has been brief
  - Disregard studies that explore UX through design, without empirical data
- Future work
  - Rich descriptions of context were surprisingly absent in the studies reviewed
  - The basic research characterizing experience over time is meagre
  - Few studies describe several aspects of UX and their interrelation
  - UX on desktop computers with productivity applications are rarely researched