In the final project of IAT 320 Body Interface, we delve into the innovative use of the human body as an input device, extending beyond traditional interfaces. This virtual game, crafted for two players, not only serves as an engaging platform but also as a means to explore and enhance social connections. Our approach integrates the sophisticated use of computer vision (OpenCV) in Python for computer vision, allowing for an immersive interaction design through camera-based tracking.
Instructor: Philippe Pasquier
Team Members:
Walter Xu ( Team Leader, Game Design Lead, Unity Programming Lead )
Baidiya Huang ( Sound Designer, QA Tester )
Delai Gao ( Environment Builder, Art Director)
Ethan Tang ( Researcher, Ideator, Programmer, QA Tester )
XingChen Cao ( Researcher, Ideator, Effect Creator )
The main data acquisition device in the system is a webcam, which is connected to the host computer via a USB interface. The task of this webcam is to acquire the motion data of the two participants. The motion data of the participants is transferred to the Python environment as soon as it is captured. We use the OpenCV library here to process the video stream.
After motion analysis, the system identifies and records key location points on the participant's body, with each participant's data consisting of 32 different points. I then combine and serialize this data and apply a smoothing algorithm on two frames to improve the motion quality. The serialized data string is transferred to the Unity software through a server address port.
Technical Workflow
Incorporating interactive elements is a pivotal aspect of our project. Navigating the complexities of 3D space, particularly about the camera's field of view, poses challenges for embedding fixed interactive elements within a two-dimensional interface. Here is one example of the many solutions.
To create a 2D collision object, I used a novel solution - the use of a cylinder with a collision volume. The cylinder was programmed in C# to fix its initial position at the midpoint of the plane shared by the two characters, and to use the position of the camera as an orientation reference. Regardless of the camera angle, the cylinder is always oriented towards the camera, rendered as a 2D circle, thus maintaining the illusion of a flat interface.
Visualization of Problem Solutions
We have designed three levels of varying difficulty, ranging from two targets in the first level to four targets in the third level. The difficulty progressively increases as two participants are required to simultaneously touch all the targets. After completing the objectives, scoring takes place, culminating in a final results page.
Upon receiving the initial draft of the map environment from our team members, I made adjustments to the sky texture and lighting. This combination of sky and lighting conveys distinct time information, transitioning from late night to dawn and finally to sunrise. Each time corresponds to a different stylistic level, providing participants with clear cues for the varying levels.
Scene Design of Three Levels
At the end of the semester an open showcase was held where I was responsible for setting up and maintaining the equipment. The class played and participated in our user test while we took notes on the players' performance.
We noticed that as the game progressed, both players showed synchronized movements even before verbal communication, highlighting the potential of games to facilitate non-verbal communication and bonding. However, we also found a significant drawback: players were often unable to make eye contact, instead focusing their attention primarily on the screen.
Commissioning Work Before The Showcase
Games can be effective in enhancing interpersonal interactions and promoting meaningful relationships. However, careful consideration must be given to the gameplay and settings to avoid total immersion of participants in the simulated environment.
It is crucial that the role of machines does not dominate and that game mechanics prioritize collaboration, communication, physical interaction and multi-player participation to achieve the goal of enhancing human interaction.
I believe that in the future of the game industry, aspects related to the use of digital media to enhance human-computer interaction will become increasingly sophisticated. Further research and projects based on these foundations will undoubtedly advance the field.
