Extended Reality

Syllabus & Schedule

• Week 1
  • Introduction
    • Course schedule and concept
    • The history of AR/VR/XR
    • Applications of AR/VR/XR
  • Overview of the XR pipeline – 3 main components of XR
    • Display technologies and rendering
    • Machine perception and input interfaces
    • Content generation – geometry, appearance, motion
  • Overview of the AR/VR/XR frameworks
  • Practical 1
    • Project structure
    • Project themes
    • Introduction to the XR programming platform
• Week 2
  • 3D scene representations and rendering
    • Object representations with polygonal meshes
    • Efficient rendering via rasterization
    • Practical XR rendering of textured geometries
  • Tracking and pose estimation for XR
    • Rigid transforms
    • Camera pose estimation
    • Body/ hand tracking
  • Practical 2

• Project draft proposal due

• Project feedback
• Rendering and displaying a first object

• Week 3
  • 3D scene capture and understanding
    • Depth estimation
    • 3D geometry capture
    • Appearance acquisition
  • Lighting estimation and relighting
    • Lighting models for 3D scenes
    • Estimating lighting in a scene
    • Relighting rendered models
  • Practical 3
    • Acquiring and utilizing depth from sensors
    • Relighting a displayed object

• Week 4
  • Physically-based simulation of rigid objects
    • Rigid-body dynamics
    • Collision detection
    • Time integration
  • Physically-based simulation of non-rigid volumetric phenomena
    • Particle systems
    • Basic fluid dynamics
  • Practical 4
• Practical due
• Project prototype due

• Project check-in
• Simulating and rendering a simple object

• Week 5
  • Advanced topics in XR I: AR/VR displays technologies
    • Micro LED-based displays
    • Waveguide-based displays
    • Next generation displays
• Week 6
  • Advanced topics in XR II – Mobile AR
    • Computer vision on limited hardware
    • UX/UI challenges
    • Content challenges

• Project due

 

Assessment

• A practical exercise, worth 20% of the mark. This will cover the basics of AR/VR development and some of the practical techniques the students will use in their projects. This is individual work. No mobile device or GPU hours are needed.
• An AR/VR project worth 80% of the marks:
  • Course projects will be designed by the students following the possible project themes proposed by the lecturer and will be checked by the lecturer for appropriateness.
  • The students will form groups of 2-3 to design, implement, report, and present the course project.
  • The final mark will be an implementation mark (40%), a report mark (20%), and a presentation/demo mark (20%). Each team member will be evaluated based on their contribution.
  • Each project will have extensions to be completed only by the ACS students. Each student will write a different part of the report, whose author will be clearly marked. Each student will further summarise her/his contributions to the project in the same report.