Sunny Zhang
Brennan Jones
Sean Rintel
ABSTRACT
The COVID-19 pandemic has raised attention toward remote and hybrid communications. Currently, one highly-studied solution lets a remote user use virtual reality (VR) to enter an immersive view of a local space, and local users use augmented reality (AR) to see the remote user's representation and digital contents. Such systems give the remote user a sense of ‘being there’, but we identify two more challenges to address. First, current systems provide remote users with limited agency to control objects and influence the local space. It is necessary to further explore the relationship between users, virtual objects, and physical objects, and how they can play a role in providing richer agency. Second, current systems often try to replicate in-person experiences, but hardly surpass them. We propose XRmas: an AR/VR telepresence system that (1) provides a multi-agency space that allows a remote user to manipulate both virtual and physical objects in a local space, and (2) introduces three family activities in a Christmas context that adopt holographic animation effects to create a ‘magical’ experience that takes users beyond merely the feeling of ‘being there’. We report on preliminary insights from the use of such a system in a remote family communication context.
Supplemental Material
- AltspaceVR. Virtual Reality Church. Retrieved February 24, 2020 from https://account.altvr.com/channels/VRChurchGoogle Scholar
- Home Assistant. Home Assistant. Home Assistant. Retrieved June 1, 2021 from https://www.home-assistant.io/Google Scholar
- Jed R. Brubaker, Gina Venolia, and John C. Tang. 2012. Focusing on shared experiences: moving beyond the camera in video communication. In Proceedings of the Designing Interactive Systems Conference, ACM, 96–105.Google Scholar
- Xiang Cao, Abigail Sellen, A.J. Bernheim Brush, David Kirk, Darren Edge, and Xianghua Ding. 2010. Understanding family communication across time zones. In Proceedings of the 2010 ACM conference on Computer supported cooperative work (CSCW ’10), Association for Computing Machinery, New York, NY, USA, 155–158. DOI:https://doi.org/10.1145/1718918.1718947Google ScholarDigital Library
- Maayan Cohen, Kody R. Dillman, Haley MacLeod, Seth Hunter, and Anthony Tang. 2014. OneSpace: shared visual scenes for active freeplay. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14), Association for Computing Machinery, New York, NY, USA, 2177–2180. DOI:https://doi.org/10.1145/2556288.2557117Google ScholarDigital Library
- Microsoft Corporation. Remote Assist | Microsoft Dynamics 365. Retrieved June 1, 2021 from https://dynamics.microsoft.com/en-us/mixed-reality/remote-assist/Google Scholar
- Omid Fakourfar, Kevin Ta, Richard Tang, Scott Bateman, and Anthony Tang. 2016. Stabilized Annotations for Mobile Remote Assistance. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16), Association for Computing Machinery, New York, NY, USA, 1548–1560. DOI:https://doi.org/10.1145/2858036.2858171Google ScholarDigital Library
- Danilo Gasques, Janet G. Johnson, Tommy Sharkey, Yuanyuan Feng, Ru Wang, Zhuoqun Robin Xu, Enrique Zavala, Yifei Zhang, Wanze Xie, Xinming Zhang, Konrad Davis, Michael Yip, and Nadir Weibel. 2021. ARTEMIS: A Collaborative Mixed-Reality System for Immersive Surgical Telementoring. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21), Association for Computing Machinery, New York, NY, USA, 1–14. DOI:https://doi.org/10.1145/3411764.3445576Google ScholarDigital Library
- Steffen Gauglitz, Benjamin Nuernberger, Matthew Turk, and Tobias Höllerer. 2014. World-stabilized annotations and virtual scene navigation for remote collaboration. In Proceedings of the 27th annual ACM symposium on User interface software and technology (UIST ’14), Association for Computing Machinery, New York, NY, USA, 449–459. DOI:https://doi.org/10.1145/2642918.2647372Google ScholarDigital Library
- Mar Gonzalez-Franco, Zelia Egan, Matthew Peachey, Angus Antley, Tanmay Randhavane, Payod Panda, Yaying Zhang, Cheng Yao Wang, Derek F. Reilly, Tabitha C Peck, Andrea Stevenson Won, Anthony Steed, and Eyal Ofek. 2020. MoveBox: Democratizing MoCap for the Microsoft Rocketbox Avatar Library. In 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), 91–98. DOI:https://doi.org/10.1109/AIVR50618.2020.00026Google Scholar
- Mar Gonzalez-Franco, Eyal Ofek, Ye Pan, Angus Antley, Anthony Steed, Bernhard Spanlang, Antonella Maselli, Domna Banakou, Nuria Pelechano, Sergio Orts-Escolano, Veronica Orvalho, Laura Trutoiu, Markus Wojcik, Maria V. Sanchez-Vives, Jeremy Bailenson, Mel Slater, and Jaron Lanier. 2020. The Rocketbox Library and the Utility of Freely Available Rigged Avatars. Front. Virtual Real. 1, (2020). DOI:https://doi.org/10.3389/frvir.2020.561558Google Scholar
- Harrie de Groot. 2021. harriedegroot/HA.Unity. Retrieved June 1, 2021 from https://github.com/harriedegroot/HA.UnityGoogle Scholar
- Jim Hollan and Scott Stornetta. 1992. Beyond Being There. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’92), ACM, New York, NY, USA, 119–125. DOI:https://doi.org/10.1145/142750.142769Google ScholarDigital Library
- Kori M. Inkpen. 2013. Kids & Video: Playing with Friends at a Distance. In Connecting Families: The Impact of New Communication Technologies on Domestic Life, Carman Neustaedter, Steve Harrison and Abigail Sellen (eds.). Springer, London, 95–123. DOI:https://doi.org/10.1007/978-1-4471-4192-1_6Google Scholar
- JBrentJ. Connect IoT data to HoloLens 2 with Azure Digital Twin and Unity - Learn. Retrieved June 1, 2021 from https://docs.microsoft.com/en-us/learn/modules/connect-iot-hololens-azure-digital-twins-unity/Google Scholar
- Brennan Jones, Yaying Zhang, Priscilla N. Y. Wong, and Sean Rintel. 2020. VROOM: Virtual Robot Overlay for Online Meetings. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA ’20), Association for Computing Machinery, New York, NY, USA, 1–10. DOI:https://doi.org/10.1145/3334480.3382820Google ScholarDigital Library
- Brennan Jones, Yaying Zhang, Priscilla N. Y. Wong, and Sean Rintel. 2021. Belonging There: VROOM-ing into the Uncanny Valley of XR Telepresence. Proc. ACM Hum.-Comput. Interact. 5, CSCW1 (April 2021), 59:1-59:31. DOI:https://doi.org/10.1145/3449133Google ScholarDigital Library
- Sasa Junuzovic, Kori Inkpen, Tom Blank, and Anoop Gupta. 2012. IllumiShare: sharing any surface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12), Association for Computing Machinery, New York, NY, USA, 1919–1928. DOI:https://doi.org/10.1145/2207676.2208333Google ScholarDigital Library
- KasaSmart.com. Smart Plugs – Kasa Smart. Retrieved June 1, 2021 from https://www.kasasmart.com/us/products/smart-plugsGoogle Scholar
- David S. Kirk, Abigail Sellen, and Xiang Cao. 2010. Home video communication: mediating “closeness.” In Proceedings of the 2010 ACM conference on Computer supported cooperative work (CSCW ’10), Association for Computing Machinery, New York, NY, USA, 135–144. DOI:https://doi.org/10.1145/1718918.1718945Google ScholarDigital Library
- David Ledo, Bon Adriel Aseniero, Saul Greenberg, Sebastian Boring, and Anthony Tang. 2013. OneSpace: shared depth-corrected video interaction. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’13), Association for Computing Machinery, New York, NY, USA, 997–1002. DOI:https://doi.org/10.1145/2468356.2468534Google ScholarDigital Library
- mattzmsft. HoloLens (1st gen) hardware. Retrieved June 1, 2021 from https://docs.microsoft.com/en-us/hololens/hololens1-hardwareGoogle Scholar
- Joshua McVeigh-Schultz and Katherine Isbister. 2021. The Case for “Weird Social” in VR/XR: A Vision of Social Superpowers Beyond Meatspace. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–10. Retrieved August 6, 2021 from https://doi.org/10.1145/3411763.3450377Google ScholarDigital Library
- Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L. Davidson, Sameh Khamis, Mingsong Dou, Vladimir Tankovich, Charles Loop, Qin Cai, Philip A. Chou, Sarah Mennicken, Julien Valentin, Vivek Pradeep, Shenlong Wang, Sing Bing Kang, Pushmeet Kohli, Yuliya Lutchyn, Cem Keskin, and Shahram Izadi. 2016. Holoportation: Virtual 3D Teleportation in Real-time. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST ’16), Association for Computing Machinery, Tokyo, Japan, 741–754. DOI:https://doi.org/10.1145/2984511.2984517Google ScholarDigital Library
- Thammathip Piumsomboon, Gun A. Lee, Jonathon D. Hart, Barrett Ens, Robert W. Lindeman, Bruce H. Thomas, and Mark Billinghurst. 2018. Mini-Me: An Adaptive Avatar for Mixed Reality Remote Collaboration. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18), Association for Computing Machinery, Montreal QC, Canada, 1–13. DOI:https://doi.org/10.1145/3173574.3173620Google ScholarDigital Library
- Thammathip Piumsomboon, Gun A. Lee, Andrew Irlitti, Barrett Ens, Bruce H. Thomas, and Mark Billinghurst. 2019. On the Shoulder of the Giant: A Multi-Scale Mixed Reality Collaboration with 360 Video Sharing and Tangible Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–17. Retrieved June 1, 2021 from https://doi.org/10.1145/3290605.3300458Google ScholarDigital Library
- Thammathip Piumsomboon, Youngho Lee, Gun Lee, and Mark Billinghurst. 2017. CoVAR: a collaborative virtual and augmented reality system for remote collaboration. In SIGGRAPH Asia 2017 Emerging Technologies (SA ’17), Association for Computing Machinery, New York, NY, USA, 1–2. DOI:https://doi.org/10.1145/3132818.3132822Google ScholarDigital Library
- Iulian Radu, Tugce Joy, and Bertrand Schneider. 2021. Virtual Makerspaces: Merging AR/VR/MR to Enable Remote Collaborations in Physical Maker Activities. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–5. Retrieved May 30, 2021 from https://doi.org/10.1145/3411763.3451561Google ScholarDigital Library
- Taehyun Rhee, Stephen Thompson, Daniel Medeiros, Rafael dos Anjos, and Andrew Chalmers. 2020. Augmented Virtual Teleportation for High-Fidelity Telecollaboration. IEEE Transactions on Visualization and Computer Graphics 26, 5 (May 2020), 1923–1933. DOI:https://doi.org/10.1109/TVCG.2020.2973065Google ScholarCross Ref
- Kimberly Tee, A. J. Bernheim Brush, and Kori M. Inkpen. 2009. Exploring communication and sharing between extended families. International Journal of Human-Computer Studies 67, 2 (February 2009), 128–138. DOI:https://doi.org/10.1016/j.ijhcs.2008.09.007Google ScholarDigital Library
- Balasaravanan Thoravi Kumaravel, Fraser Anderson, George Fitzmaurice, Bjoern Hartmann, and Tovi Grossman. 2019. Loki: Facilitating Remote Instruction of Physical Tasks Using Bi-Directional Mixed-Reality Telepresence. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST ’19), Association for Computing Machinery, New York, NY, USA, 161–174. DOI:https://doi.org/10.1145/3332165.3347872Google ScholarDigital Library
- Nadir Weibel, Danilo Gasques, Janet Johnson, Thomas Sharkey, Zhuoqun Robin Xu, Xinming Zhang, Enrique Zavala, Michael Yip, and Konrad Davis. 2020. ARTEMIS: Mixed-Reality Environment for Immersive Surgical Telementoring. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA ’20), Association for Computing Machinery, New York, NY, USA, 1–4. DOI:https://doi.org/10.1145/3334480.3383169Google ScholarDigital Library
- John P. Whitney, Tianyao Chen, John Mars, and Jessica K. Hodgins. 2016. A hybrid hydrostatic transmission and human-safe haptic telepresence robot. In 2016 IEEE International Conference on Robotics and Automation (ICRA), 690–695. DOI:https://doi.org/10.1109/ICRA.2016.7487195Google ScholarDigital Library
- Svetlana Yarosh, Anthony Tang, Sanika Mokashi, and Gregory D. Abowd. 2013. “almost touching”: parent-child remote communication using the sharetable system. In Proceedings of the 2013 conference on Computer supported cooperative work (CSCW ’13), Association for Computing Machinery, New York, NY, USA, 181–192. DOI:https://doi.org/10.1145/2441776.2441798Google ScholarDigital Library
- 2021. microsoft/Virtual-Robot-Overlay-for-Online-Meetings. Microsoft. Retrieved June 10, 2021 from https://github.com/microsoft/Virtual-Robot-Overlay-for-Online-MeetingsGoogle Scholar
- 2021. microsoft/MoveBox-for-Microsoft-Rocketbox. Microsoft. Retrieved June 1, 2021 from https://github.com/microsoft/MoveBox-for-Microsoft-RocketboxGoogle Scholar
- 2021. microsoft/Microsoft-Rocketbox. Microsoft. Retrieved June 1, 2021 from https://github.com/microsoft/Microsoft-RocketboxGoogle Scholar
- Spatial: Virtual Spaces That Bring Us Together. Retrieved June 1, 2021 from https://spatial.io/Google Scholar
- Introducing Microsoft Mesh | Here can be anywhere. Retrieved June 1, 2021 from https://www.microsoft.com/en-us/meshGoogle Scholar
- Hubs by Mozilla. Retrieved June 1, 2021 from /Google Scholar
- VRChat. VRChat. Retrieved June 1, 2021 from https://hello.vrchat.comGoogle Scholar
- Product | RICOH THETA V. Retrieved June 1, 2021 from https://theta360.com/en/about/theta/v.htmlGoogle Scholar
- Guardian® GT Force-Multiplying Dexterous Robotic System. Sarcos Robotics. Retrieved June 1, 2021 from https://www.sarcos.com/products/guardian-gt/Google Scholar
- Digital Twins – Modelling and simulations | Microsoft Azure. Retrieved June 1, 2021 from https://azure.microsoft.com/en-ca/services/digital-twins/Google Scholar
- Avatar Maker Pro - 3D avatar from a single selfie | Modeling | Unity Asset Store. Retrieved June 1, 2021 from https://assetstore.unity.com/packages/tools/modeling/avatar-maker-pro-3d-avatar-from-a-single-selfie-134800Google Scholar
- Oculus Quest 2: Our Most Advanced New All-in-One VR Headset | Oculus. Retrieved June 1, 2021 from https://www.oculus.com/quest-2/?utm_source=bing&utm_medium=d_ps&utm_campaign=407133826&utm_term=oculus+quest+2&utm_content&utm_parent=quest2&utm_ad=1363394943304446&utm_location=5044&utm_location2&utm_placement=kwd-85212471786018%3Aloc-4056&utm_adposition&utm_device=c&utm_matchype=e&utm_feed&msclkid=4f115a46b98d1813d7a23af04672a49fGoogle Scholar
Recommendations
Merging environments for shared spaces in mixed reality
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and TechnologyIn virtual reality a real walking interface limits the extent of a virtual environment to our local walkable space. As local spaces are specific to each user, sharing a virtual environment with others for collaborative work or games becomes complicated. ...
Experiencing Social Augmented Reality in Public Spaces
UbiComp/ISWC '21 Adjunct: Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable ComputersWe present various views on the future use of augmented reality in public spaces. The views address enhanced walking, social activity, appropriation of public spaces, and futuristic social aspects of future outdoor augmented reality. Although we often ...
Definition of a Reference Architecture for the Integration of Augmented Reality/Virtual Reality Techniques in Remote Experimentation Environments
Proceedings TEEM 2022: Tenth International Conference on Technological Ecosystems for Enhancing MulticulturalityAbstractRemote labs are especially important for learning because they bring real results and each experience is unique. On the other hand, lack of immersion is one of the negative aspects of remote labs. This aspect can be solved through the integration ...
Comments