When XR and AI Meet - A Scoping Review on Extended Reality and Artificial Intelligence

Author

Teresa Hirzle, Florian Müller, Fiona Draxler, Martin Schmitz, Pascal Knierim, Kasper Hornbæk, CHI 2023

extended reality, artifcial intelligence, scoping review

summarize the state-of-the-art XR and AI research by presenting a typology including fve main topics
present an overview of algorithms, tools, datasets, models, data types, and user study data from the reviewed papers
critically discuss current research gaps, and provide 13 research opportunities, as well as recommendations for future work

Research Questions
- RQ1: What are the main topics researched at the intersection of XR and AI?
- RQ2: What are the main problem areas that are addressed with XR and AI research?
- RQ3: How is the research conducted? In particular, what algorithms, tools, datasets, models, data types, and user study data are employed to conduct the research?
Search Strategy
- Definition of keywords
- Search
Evidence Screening and Selection
- Exclusion Criteria
  - Not in main proceedings
  - Survey or literature review
  - Year
  - Missing term
  - False positive
  - Example mension
  - Example application
  - Dataset
  - Lacking information
- Inclusion Criteria
  - AI method applied for XR
  - XR applied for AI
  - Interaction with embodied AI
  - Application focus
  - Requires further reading
Data Extraction and Code Book
Critical Appraisal, Potential Bias, and Limitations
Analysis and Informal Synthesis

Overview of Papers
- Research Directions
- Publication Venues
- Distribution of XR Technologies
- Distribution of Keywords
Typology of the State-Of-The-Art XR and AI Research
- Using AI to Create XR Worlds
  - Creating environment
  - Creating avatars
  - Creating agents
  - Creating objects
- Using AI to Understand Users
  - Predicting VR Sickness
  - Predicting User Characteristics
  - Predicting Viewport and Head Movement
  - Eye Tracking and Gaze Analysis
- Using AI to Support Interaction
  - Gestural Interaction
  - Locomotion Techniques
  - Novel Devices
  - Novel Interaction Techniques
  - Haptic Feedback
- Interacting with Intelligent Virtual Agents
- Using XR to Support AI Research
- Topic Distribution for HCI Papers
Main Problem Areas Addressed in XR and AI Research
- perception and neuroscience
- interacting with IVAs
- presentation of virtual content
- tracking technologies
- health in XR
- creating high fdelity human characters
- interaction techniques
- social and ethical issues
- building devices
- rendering
- display technology
Algorithms, Tools, Datasets, Networks, Data Types, and User Study Data
- Algorithm Techniques and Classes
  - Algorithm Techniques
  - Algorithm Classes
- Tools, Datasets, and Networks
- Data Types
- Assessment and Evaluation
- When is AI applied?