Module 1: Introduction

Virtual Reality (VR) is not just a modern technological novelty; it is the result of years of research, experiments, and innovations. In this module, we will embark on an exciting journey through the history of VR, from its earliest ideas and concepts to today's groundbreaking technologies. We will explore key milestones in the development of VR, significant theoretical approaches, and meet the pioneers who laid the foundations of this field. You will discover how virtual reality has changed our perception of the world and opened new possibilities for science, education, medicine, and entertainment. Ready to begin? The immersion starts now!

1.1 Historical Background & basics

Virtual Reality (VR) is not a new concept but rather a culmination of ideas and technologies developed over several decades. Its roots can be traced back to panoramic paintings and early mechanical simulators, but the digital age has accelerated its evolution substantially.

1.1.1 Pioneering Days: 60-70s:

The first known system to convey the illusion of reality was developed by the cinematographer Morton Heilig in 1956 and patented in 1962. He called the first prototype of a multi-sensory simulator "Sensorama". The virtual reality machine was quite a large booth, in which film projectors were integrated, reproducing movies on a stereoscopic screen. It also featured stereo speakers (reproducing, for example, an audio recording of the city noise), a vibrating seat for simulating vibration (like, for example, in a sinking ship or rockfall), a system for simulating various smells and an emulator of atmospheric effects (such as wind using a hair dryer and rain).

According to the creator of the device, Heilig, Sensorama was supposed to become the future of the film industry. But, as we know, it did not happen that way. The machine remained a controversial attraction, for which only six short films were developed. The booth was cumbersome and expensive, which damaged Sensorama's chances of big investment and scaling-up.

Figure 1 - Sensorama[1]

In 1961, Philco Corporation constructed the first head-mounted display called Headsight4 for military purposes, and this became the first application of the technology in real life. The device, which later became widespread, came to life as a top-secret military project. Philco Corporation engineers integrated a video display and a basic tracking system into a helmet and connected everything to CCTV. The head movements were transmitted to the camera drives and allowed you to observe in the helmet what the camera was directed at. Headsight was developed to enable remote viewing of situations that are too dangerous to be in close proximity of. But based on the current classification, the system would be better regarded as AR technology.

Figure 2 - Headsight

In 1963, writer and inventor Hugo Gernsback created mass-produced television eyeglasses (Hugo Gernsback). They still did not track head movements and were not designed for work with computers. The television eyeglasses made it possible to receive TV shows and display TV content (including stereoscopic images) using two small cathode-ray tubes. In the 1960s, the aerials protruding from the device conjured up images of space and even aliens in customers. Unlike previous devices, the television eyeglasses were mass-produced.

Figure 3 - Hugo Gernsback

The iconic red stereoscope blended several different images of the same scene creating a single 3D image. The device was the first one to create a feeling of immersion into another world, which has now become familiar to everyone.

Figure 4 - Viewmaster

The American scientist Ivan Sutherland, better known as one of the "fathers of the Internet", is rightly regarded as a pioneer of computer-generated VR technology. In 1965, Sutherland presented the concept in which a computer simulation of the real world affected the user with the help of a special helmet creating such a realistic illusion, so that a person would not be able to differentiate the simulation from actual reality. The concept also included the user being able to interact with objects in virtual reality. The possibilities described in his paper were later perceived by researchers and developers as a goal, the concept began to be considered as a fundamental blueprint for virtual reality. Sutherland saw the peak of technology development in the sense that the user could compare the experience of using VR with Carroll's Alice's journey to Wonderland.
In 1968, Sutherland, together with his student and colleague Bob Sproull, developed the first virtual reality head mounted display system. It was named the "Sword of Damocles", due to its fixed system of strapping. The device, by current standards, was quite primitive, as it displayed only simple virtual wire-frame shapes in 3D. The "Sword of Damocles" was equipped with a head movement tracking system, when the user moved their head, the 3D models on the screen changed their perspective (visual feedback). The difficulty was that all the components of the system were so heavy that the device had to be suspended from the ceiling on a mechanical arm for a user to wear, and a picture was transmitted through the computer. Despite the device being quite cumbersome, the CIA and NASA became interested in the technology.

Figure 5 - Sword of Damocles

The next step in the development of virtual reality technologies is Kruger's VIDEOPLACE. The complex is the first interactive VR platform. The system was first shown at the Milwaukee Arts Center in 1975. VIDEOPLACE used computer graphics, projectors, video cameras from one of the earliest tracking systems for object location. The system, like the military simulators of the time, did not use glasses or gloves.

The platform was based on a dark room with large screens that surrounded the user. Like modern kinect systems, the user's movements were transmitted to computer models. Also, the models of users in different rooms could interact with each other in a virtual space, which was a fundamentally new function for systems of this kind.

Figure 6 - Videoplace

Another ideological and conceptual rather than technological breakthrough was the film map of Aspen, Colorado, created by a team of researchers from Massachusetts Institute of Technology.  The system, introduced in 1977, allowed for a virtual tour of the city, much like Google Street View does.

The content was created by repeatedly photographing the city with a car driving through the streets. During the virtual tour, the footage was interactively reproduced during a virtual visit to the city. A helmet was not used for this, standard CRT and projection TVs were used. The conceptual breakthrough was that VR technologies can simulate other places with sufficient accuracy.

Figure 7 - Aspen Movie Map

In 1977 Daniel J. Sandin and Thomas DeFanti created the first wired data glove based on an idea by Richard Sayre as a project for the National Endowment for the Arts. It used light based sensors with flexible tubes with a light source at one end and a photocell at the other. As the fingers were bent, the amount of light that hit the photocells varied, thus providing a measure of finger flexion. It was a lightweight, inexpensive effective method for multidimensional control, mainly used to manipulate sliders.

Figure 8 - Sayre Glove

In 1979, the first VR helmet appeared that was close to modern functionality. We are talking about a small-scale model of the VITAL military flight simulator developed and marketed by the McDonnell-Douglas Corporation10. It was equipped with a head and eye tracking system, as well as with stereoscopic vision glasses for computer-generated images. The helmet featured displays for each eye, stereo headphones and a microphone for communication with the flight dispatcher.

Figure 9 - The first HMD

1.1.2 The ’80-90s

Pioneer and wearables "daddy" Steve Mann created an unwieldy backpack computer connected to a helmet-cam and viewfinder. It harnessed a beam splitter to send a scene both to the user and to a computer-connected camera, allowing the overlay of real-time data. Although Mann's reality is augmented and not virtual, his subsequent prototypes refined Eye Tap and showed that virtual technologies needn't be bulky and unnatural.

Figure 10 - Steve Mann

Reality Built for Two was the first commercial VR system from the company set up by VR legend Jaron Lanier included the DataGlove, which allowed users to twist and turn virtual objects that appeared in the EyePhone heads-up display. However, you needed deep pockets as the whole package would set you back $100,000, although there was a budget option available for $50,000.

Figure 11 - RB2 - First VR controllers

в

The space agency has been instrumental in keeping VR alive over the last four decades, combining LEDs, liquid crystal displays and wide-angle optics to good effect. It also pioneered head tracking technology that was built upon by successive efforts from private companies. 

The National Agency used virtual reality devices that were well-known at that time. One of the innovations was the use of a full-fledged suit, as well as an audio system, which was promising in terms of its realistic sound simulation. A special feature of the latter was the use of binaural effects instead of banal stereo; it was this system that Foster's company was working on.

Figure 12 - NASA – LCD optics & head tracking

The Japanese company Sega was the first to make their debut with the Genesis VR gaming console17. In 1987, SegaScope 3-D Glasses (1987) were released, therefore, many call Sega the pioneers of VR in the gaming industry, which is not quite true. The fact is that SegaScope 3-D, developed by Mark Cerny, the designer of the game Marble Madness, became an accessory for the Sega Master System18.  The glasses created the illusion of 3D graphics in games, which made a great impression on players back in the times when 3D was something exceptional. At the same time, SegaScope 3-D was not exactly a VR device in every sense of this term, since it did not create an imitation of the user's presence in virtual space, did not track their movements and were limited to creating three-dimensional images. Despite this, the success of games such as Maze Hunter, Missile Defense and Zaxxon 3D heightened interest in virtual reality. The original principle of the glasses is also interesting. The TVs to which the gaming console was connected displayed different images to each eye via the glasses (shutter glasses) in quick succession, so that a 3D effect was simulated. There was also a negative effect, due to this principle, the frame rate was reduced by 2 times and it was perceived by users as blinking.

Unfortunately, a lukewarm reaction to the platform had SEGA claiming that the experience was too real and that gamers could injure themselves. Very quietly, the project was mothballed.

Figure 13 - SEGA VR

ALIVE system, used computer and video overlays so users and virtual agents could interact. One of the first operational gesture-based systems developed was the ALIVE system developed in 1993 at MIT Media Laboratory. It used computer and video overlays that enabled the user and “virtual agents” to interact in same display space.

Figure 14 - ALIVE

The Cave Automatic Virtual Environment (CAVE) was invented by students of the University of Illinois. They managed to develop lightweight stereoscopic LCD shutter glasses similar to those that are produced today. The CAVE technology allowed for the creation of a three-dimensional space where users could walk around and through.  Another innovative feature of CAVE was that several users could be in the virtual space at the same time.

Figure 15 - CAVE

1.1.3 Modern Era

If it wasn't for the launch of the crowdfunding platform, it might have taken Oculus VR founder Palmer Luckey a lot longer to get his virtual vision made real.

The real boom began in 2012. On August 1, 2012, a little-known startup Oculus initiated a Kickstarter campaign to raise funds for the release of a virtual reality helmet named the Rift. The developers promised users a "full immersion effect" through the use of displays with a resolution of 640 by 800 pixels for each eye. The necessary amount of 250 thousand dollars was raised in the first four hours. If the Crowdfunding campaign hadn't been launched, perhaps the founder of Oculus VR, Palmer Lucky, would have taken much longer to realize his vision of virtual reality. Oculus Rift raised almost $ 2.5 million from 10, 000 contributors, thereby creating the most successful story of the Crowdfunding platform.

Since 2015, virtual reality technologies have become a real honeypot for technological companies.

World-famous companies have been actively involved with the development of immersive technologies.  The owner of Universal Studios, Comcast, invested $6.8 million in a small VR studio Felix & Paul in Montreal, which collaborated with Funny or Die and the White House. The New York Times has also been investing in the development of virtual reality. For many years, the media outlet has been creating 360-degree videos that have won awards at the Cannes Lions Festival.

In 2014, Facebook purchased Oculus VR for $2 billion, and in 2017 launched a social network featuring 3D avatars and VR interaction.

In 2016, HTC began developing its Vive platform. HTC develops the HTC RE product series and their virtual reality helmet has long been a competitor to the Oculus. HTC not only develops its products, but also creates accelerators for VR startups. 

Other major market players also do their part. In 2015, Sony acquired SoftKinetic Systems, a developer of gesture recognition solutions, and Samsung announced an investment of millions of dollars in the American startup Baobab Studios, specializing in animated virtual reality.

Large companies produce their own virtual reality headsets (HTC Vive, Oculus, PlayStation VR), while developing exclusive games and software only to be used with their products. With the aim of cooperation, big names of the industry established the Global Virtual Reality Association (GVRA) at the end of 2016 to promote development and adoption of VR. The association includes Acer (Starbreeze), Google (Cardboard, Daydream), HTC (Vive), Facebook (Oculus Rift), Samsung (Gear VR) and Sony (PlayStation VR) and others.

Figure 16 - Apple vision pro

Apple unveils the Vision Pro, a groundbreaking mixed reality headset, marking its foray into a new product category since the Apple Watch in 2015. Introduced at WWDC 2023 and set for early 2024 release, the Vision Pro is more than a headset; Apple calls it a "spatial computer. "Features a non-see-through display, utilizing cameras to blend AR content with the physical world, while VR content is fully immersive. Switching between AR and VR modes is seamless with the Digital Crown, akin to changing the world with a twist. The Vision Pro is equipped with over 4K micro-OLED displays per eye and an EyeSight external display to convey user's eye presence. For input, it ditches traditional controllers for eye tracking, hand gestures, and voice commands, revolutionizing interaction. Optic ID technology offers secure authentication, similar to Face ID or Touch ID, but with iris scanning. Powered by M2 and a new R1 chip, it supports a dedicated operating system, visionOS, with an App Store for bespoke apps and experiences.

The Apple Vision Pro headset is a comprehensive mixed reality device that combines elements of both virtual reality (VR) and augmented reality (AR). Unlike AR glasses that are typically transparent, the Vision Pro covers your eyes completely, creating a digital representation of your environment rather than directly viewing it.

How it operates:

Mapping Real-World Environments:

• Exterior cameras on the headset scan and map the surrounding space.
• Interior cameras track the user's eye and hand movements for interactive experiences.

Projection of the Real World:

• The headset doesn’t use transparent displays; instead, it captures the real-world using cameras and then recreates it within the headset.
• This allows for a digital overlay of 3D content and objects within the user's environment.

Switching Between Realities:

• Users can choose to have augmented reality content integrated into their real-world view.
• Alternatively, they can switch off the exterior cameras to engage in a fully virtual experience, isolating themselves from the physical world.

Flexibility in Use Cases:

• The headset is designed to support a variety of activities, including work-related tasks, media consumption, gaming, and communication.
• Apple’s aim with the Vision Pro is to enable these activities to be carried out in any location, promising a high degree of mobility and flexibility.

The Vision Pro's advanced technology signifies Apple's commitment to creating a versatile device that can adapt to the user's needs, whether they are looking to interact with virtual objects in their actual room or immerse themselves in a completely virtual environment. This adaptability and multifunctionality are central to Apple's concept of working and entertaining "anywhere" using just the Vision Pro.

Figure 17 - Milestones in the history of VR (Molnár, 2016, p. 105)

1.2      Difference between VR and AR/XR/MR

While often lumped together in conversations about immersive technologies, Virtual Reality (VR), Augmented Reality (AR), Extended Reality (XR), and Mixed Reality (MR) are distinct in their functionalities and applications. This section elucidates the differences and commonalities among these technologies, offering a more nuanced understanding of each.

1.2.1 Core Objectives: Immersion vs. Overlay

In essence, VR seeks to create a completely immersive experience by cutting off the real world and substituting it with a virtual one. In contrast, AR and MR aim to augment the      physical world by overlaying digital elements or holograms. Here, the objective is not complete immersion but rather an enhanced or "augmented" reality.

Figure 18 Differences between VR, MR and AR

1.2.2.Defining AR, MR, and XR

• AR (Augmented Reality): Defined as the projection of digital information and holograms into the      immediate environment, AR does not      replace the real world but complements it. AR is commonly used in applications ranging from navigation aids to interactive 3D ads.
• MR (Mixed Reality): MR takes AR a step further by allowing for interactions between the real and virtual elements. For instance, one      could manipulate a virtual object using a real-world tool.
• XR (Extended Reality): This is an umbrella term that includes all immersive technologies like VR, AR, and MR. XR is the spectrum of experiences that link and transition between the completely real and the completely virtual.

1.2.3 The Role of Immersion

Immersion serves as a fundamental aspect for both VR and AR, enabling users to deeply engage with content. Immersive experiences can range from simple 360-degree videos to complex interactive simulations. This is why VR and AR are often categorized under the broader term of "immersive media."

1.3     Hardware Differences

1.3.1 Technological Interplay in MR Glasses

Mixed Reality glasses epitomize the convergence of AR and VR technologies. These devices not only allow for full immersion into a virtual world but also employ cameras, sensors, and spatial computing algorithms to capture the real environment. This enables a seamless integration of virtual content into the user's physical space, fulfilling the criteria of both AR and MR.

1.3.2 VR and AR glasses (goggles)

VR Devices: Typically uses head-mounted displays (HMDs) as output devices, like Oculus Rift or HTC Vive, which often come with external sensors for tracking movements.

AR Devices: While AR can also be experienced through HMDs like Microsoft's Hololens, it is often accessible through more conventional devices such as smartphones, tablets, or PCs.

MR Devices: These are specialized glasses or headsets that combine the features of both VR and AR. They use advanced tracking technologies to understand the spatial context of a room, allowing them to overlay virtual objects onto the real world in a way that they interact with the user and each other.