VRySmart
The concept of this work is to design and build a working framework that would help developers to integrate smartphones into virtual and mixed reality.
The screenshot is from a real VR application
Motivation:
Smart devices are ubiquitous
Smart devices are everyday objects
Smart devices can act as fully-fledged controller
Smart devices have rich functionality that can be utilized in VR
Users are familiar with the interaction style and affordances
The framework aims to improve experience in following aspects:
Haptic feedback
Interaction
Notifications handling
Functionality
The framework handles following functionality:
Networking
Marker based Positional tracking of the device
Translation of touch-events from the device to the main app
Vibration calls from the main app to the device
Camera streaming (based on Unity Render Streaming)
Hands/body Segmentation (based on SRWorks)
Body segmentation (ML-based)
Future work:
SLAM-based tracking
Screen content streaming
Use cases
Passive
Functional
Logical
Social