Course Information

Course Description

All course materials are provided in Moodle

Course Description

Brief Description :

The goal of this course is an introduction and hands-on experience on latest mixed reality technology at the cross-section of 3D computer graphics and vision, human machine interaction, as well as gaming technology.

Learning Goals :

After attending this course, students will:

  1. Understand the foundations of 3D graphics, Computer Vision, and Human-Machine Interaction
  2. Have a clear understanding on how to build mixed reality apps
  3. Have a good overview of state-of-the-art Mixed Reality
  4. Be able to critically analyze and assess current research in this area
Course Evaluation :
  1. 5 %: Course Participation (e.g., during the class in lectures and paper discussion or at the online forum)
  2. 25%: Paper presentation (incl. discussion moderation)
  3. 70%: Final project which includes a report and presentation/demo
Prerequisites / Notice :

Prerequisites include:

  • Good programming skills (C# / C++ / Java etc.)
  • Computer graphics/vision experience: Students should have taken, at a minimum, Visual Computing. Higher level courses are recommended, such as Introduction to Computer Graphics, 3D Vision, Computer Vision.
Related Courses :
Course Schedule :
No. Date Topic Presenter Paper Presentations
1. 27.09 MR Introduction Iro Armeni, Federical Bogo -
2. 04.10 Introduction to Unity, HoloLens, and ASA Patrick Misteli (Microsoft) -
3. 11.10 Project Proposal Presentation - -
4. 18.10 Guest Lecture Martin Oswald 2, 10, 16
5. 25.10 Mixed Reality with HoloLens 2 Marc Pollefeys 1, 8
6. 01.11 Experience Design for Mixed Reality - how to create empowering experiences Stella Muehlhaus (Facebook) 7, 22, 9
7. 08.11 Augmented Virtuality, Diminished Reality, Substitutional Reality Andreas Fender (ETH) 4, 14, 15
8. 15.11 Mid-Term Project Presentation - -
9. 22.11 3D scene understanding for AR applications Federico Tombari (TUM/Google) 3, 20, 21
10. 29.11 Medical AR Esfandiari Hooman (Balgrist UniversitatsKlinik) 5, 6, 13
11. 06.12 Virtual Reality and Virtual Humans for Psychological Research: Challenges and Opportunities Stephan Streuber (Univ. Konstanz) 12, 18
12. 13.12 Egocentric perception and anticipation for user-centric applications; Incon.ai: Instructive Construction Antonino Furnari, Giovanni Maria Farinella (University of Catania); Timothy Sandy (ETH) 11
13. 20.12 Final Project Presentations - -
Paperlist :
  1. Depth from motion for smartphone AR
  2. CoolMoves: User Motion Accentuation in Virtual Reality
  3. MEgATrack: Monochrome egocentric articulated hand-tracking for virtual reality
  4. Human touch: Social touch increases the perceived human-likeness of agents in virtual reality
  5. Egocentric pose estimation from human vision span
  6. Passthrough+: Real-time stereoscopic view synthesis for mobile mixed reality
  7. Evaluating grasping visualizations and control modes in a VR game
  8. Driving signal aware full-body avatars
  9. DepthLab: Real-time 3D interaction with depth maps for mobile augmented reality
  10. Deep implicit volume compression
  11. TapID: Rapid touch interaction in virtual reality using wearable sensing
  12. SMPL: A Skinned Multi-Person Linear Model
  13. MRTouch: Adding Touch Input to Head-Mounted Mixed Reality
  14. HoloDesigner: A mixed reality tool for on-site design
  15. LoBSTr: Real-time lower body pose prediction from sparse upper-body tracking signals
  16. DeepFovea: Neural reconstruction of foveated rendering and video compression using learned statistics of natural videos
  17. SuperGlue: Learning feature matching with graph neural networks
  18. Real-time mixed reality teleconsulation for intensive care units in pandemic situations
  19. Mixed reality light fields for interactive remote assistance
  20. LoFTR: Detector-Free Local Feature Matching with Transformers
  21. H2O: two hands manipulating objects for first person interaction recognition
Deadlines :

October 02: Group formation and project selection - Students select from a list of project proposals and we assign them to the topics.
October 10: Project proposal documents - Students submit their project proposal documents after discussing with their assigned supervisors.
October 11: Proposal presentations - Students present their project proposals during lecture.
November 15: Midterm presentations - Students present their progress on their projects during lecture.
November 16: Midterm project report - Students submit a written report with current updates and plan.
December 20: Final project presentations - Students present their projects in a joint session.
January 07: Final project reports - Students submit their final reports for the projects.
* All the submission deadlines are due by 23:59 on the specified date.

Student Projects :

Over the semester, students will work on a project related to a topic in Mixed Reality in collaboration with a supervisor. Students are required to form groups of 4 and submit their preferred project topics first. For this purpose, we will provide a list of project suggestions, but you are free to propose your own project.
For your projects, we can make the following hardware available:

  • Microsoft HoloLens
  • Mobile phone devices
    Students with project idea that require any other special equipment, please talk to us! You will also get access to a Windows machine that you can use for development purposes.
    Each student group is then required to hand in and present a project proposal by the announced deadline. Make sure to talk to your assigned supervisor and discuss the project with him/her while planning your proposal. Each team will present their project proposal during a designated lecture. The template for the project proposal report can be found here.
    The proposal should be 1-2 pages describing what you want to do in the project, and how you plan to achieve your envisioned results. A good idea is to identify the algorithmic and technical challenges within the project. Try to address each of them individually and explain your considered solutions; also make an attempt to think about alternatives if you believe a particular approach is unstable or likely to fail.
    Midterm presentations have the purpose that you present what you did so far and that you get feedback. So you are encouraged to raise open questions. This is a possibility for us to steer the project and help you, if you got stuck. In addition, you are required to submit a 2-page report that identifies your current status, problems you have identified, and your next steps.
    Final presentations will be held as a regular presentation session. In addition, you are required to hand in a technical report for your project. The report format should be in parallel with CVPR paper format. Latex and Word templates can be found here.
Student Paper Presentations :

In each class, a (guest) lecture will be given first. After the lecture, students, together with their project group members, will give presentations on selected papers.
Other students are encouraged to engage in the paper presentations through active discussions. To organize the discussion in a more lively way, each project group will be assigned to lead the discussion of an other project group's presentation; i.e. this project group acts as an "opponent" or moderator and actively supports the discussion by asking relevant questions wrt. the presented paper and motivates other students to contribute.
We will assign each group a paper and a presentation date after the projects are assigned. The list of papers to be presented by students will be published soon.

Discussion Forum :

There exists a discussion forum page in MOODLE for this course. Students should sign in using their ETHZ accounts and participate in the discussion forums. Please put all your discussions related to the lectures, paper presentations and projects there.

Some useful links :

How to read and present scientific papers

Semester Index