Virtual Reality and Worldbuilding

Cover of "Matrix-Trilogy [Blu-ray]"
Cover of Matrix-Trilogy [Blu-ray]
Virtual reality (VR) is quite familiar in science fiction, with The Matrix one of many stories to make use of the trope. But is it plausible? To my engineer’s view, the short answer is yes. But as when we looked at teleportation or whether dragons can really fly, there are challenges. As always, these can simply be overlooked since VR is accepted in the genre but by working through the issues you may find ways to offer the reader a unique take on VR.

First off, there are really two basic types of VR: organic entities interfaced to a virtual world and virtual entities in a virtual world. These two types serve to define some of the challenges facing VR but they need not be exclusive. For instance, in The Matrix, you have humans and AIs like Agent Smith both in the same VR. Let’s start with organic entities because in some ways they actually present greater challenges.

U.S. Navy personnel using a VR parachute trainer
U.S. Navy personnel using a VR parachute trainer (Photo credit: Wikipedia)

You can buy VR games with VR goggles today. Or take a trip in a “VR” ride at an amusement park or local zoo. But in the end, they aren’t terribly satisfying. Why is that? Primarily because they provide limited sensory replacement. Sound immersion is probably the best of the VR senses these days. Vision does pretty well but a monitor does not provide any peripheral vision and goggles have yet to really provide the equivalent of what an eye ball darting all around can see (plus they are heavy). There are some gloves that provide both manipulation and some tactile feedback but this is, in the end clumsy. No system can provide something as simple as the feel and heft of a baseball from the smooth leather with occasional scuffs to the stitching, let alone the feel of the mass in your hand. Of course, there is also taste and smell, which are very difficult to model, let alone the tactile sense of something in your mouth.

But these are the obvious five senses. What about sense of balance with its automatic sense of acceleration and level? What about forces felt throughout your body: a jarring landing from a jump off a ladder, acceleration in a car, centripetal forces in a turn. Your VR ride can get you some sense of the motion. There is a kickstart project for a VR treadmill so you can even feel like you are moving. But in the end, while all the present techniques may help you immerse yourself in a world,  you would never mistake such a VR world for reality. Too many things would not mesh.

Instead, you need something like what they did in The Matrix: direct connection of the nervous system, both out-bound and in-bound. In the movie, everything was interfaced, which meant that the simulation could provide every possibly sense from the conventional 5 senses to sensations throughout the body. With this system and a sufficiently advanced simulation, a human would not know they were in a virtual world: they could literally be a “brain in a jar” and would never know it.

In summary, the verisimilitude of a VR simulation is going to depend on the amount of sensory information fed directly to the brain. Work through our existing senses, as we do with current technology, and the VR experience will be rather limited, though it might still be a lot of fun. Start “jacking” in directly into our nervous system and the experience can improve. Provide full sensory replacement and theoretically, it should be indistinguishable. And has been done often in fiction, nothing says the sensory jacks need be permanent. It ought to be possible to switch between your body’s sensory network and a computer one. (Regarding The Matrix, though, why would the machines have made it work that way?)

Let’s turn to providing all this sensory information. It is a fairly large amount of information although, in the end, stimulating the retina with its 160M or so receptors + all the other sensors, which is much less information than what the visual cortex processes, is do-able. It’s a lot of information but 3D displays are already within a few orders of magnitude of it and there are tricks focusing on the main field of vision that could reduce it. The hard part is not providing the sensory representation of the particular VR world state at the moment. It is modeling that VR world in a realistic way to provide that state.

Imagine this: you are on the beach. You grab a handful of sand and pour it through your fingers. There’s probably one million grains of sand just in your handful. You can imagine how many more are on the one beach, let alone all the beaches of the world. Does your VR engine model every grain of sand? Probably not. But let’s say you look closely at that crust of sand left over on the skin of your hand. Now you can see individual grains of sand, tens of thousands. Although you don’t, of course, look at each one individually, who is to say where you will focus, which means the computer will have a challenge knowing when to provide the extra resolution, when to resolve the sand as grains rather than a mass of sand.

In The Matrix, there’s the joke about why everything tastes like chicken: that’s what the computer uses when it isn’t sure what something tastes like. In the world we are familiar with, something as simple as sand comes in many sizes, many degrees of wear, many mineral compositions. There are scientists who study sand and they can read a great deal about where it came from and how it came to be there. How much of that does the computer model? Does it make all sand the same (“chicken sand”) or does it vary the composition from monazite sand, source of rare earth elements, to coral sand, volcanic sand, etc., etc. Maybe the average person wouldn’t really notice but some would. And if you don’t happen to know or care much about sand, what about your own areas of expertise related to your hobbies and professions? A wood-worker will know the properties of many types of wood. A saltwater fish enthusiast will expect his VR aquarium to model the real world, and so on.

Modeling the real world at a level of complexity sufficient to fool a large number of organic entities seems almost impossible. Perhaps it could work for a limited number of organic life forms, say the crew on a spaceship on a hundred year voyage. But for billions of humans? The computational requirements seem essentially impossible: is it plausible to have sufficient compute resources located in sufficient proximity to the billions of organics to provide the responsiveness necessary to fool human brains living in the present? I think not. (And did I mention I’m a computer engineer?)

But before we give up on it, there is one avenue of approach that would likely work and that is to change the level of abstraction of what the computer is feeding your brain. Let’s take vision. Your retina has about 150 million receptors but that’s not what gets sent down your optic nerve. Your retina has a thin layer of neurons that does some pre-processing that already reduces the information to your visual cortex by about two thirds. Once this information gets to your visual cortex (which is a sizable portion of your brain so you can tell just how complicated this processing is) there are further layers of processing, each abstracting the information you digest until you aren’t literally digesting the curls and lines of the letters of this blog post, you aren’t even really seeing the individual letters. Your elegant brain has taken the information from 150 million receptors and reduced it to the elements of language. It does the same for sound, even taste, smell, sense of balance.

If the “jacks” into your awareness are “higher up the stack”, if the VR simulation is feeding you higher levels of abstraction then computational requirements for the VR simulation begin to drop dramatically. It never has to feed you an image of grains of sand. It just has to tell you that you are looking at grains of sand. This type of jack, however, would need to be implanted deep within your brain. It seems unlikely (but perhaps not impossible) for it to be of a type that could switch between VR stimulation and real sensory stimulation. But maybe, in some cases, there would be no need to switch between the two: what about an organic race that has abandoned bodies? Or a soldier of the 22nd century whose body was destroyed in battle? Or perhaps a combat pilot of the 23rd century who forgoes his body so that he can interface with high G drones? Lots of dramatic potential here and lots of opportunity to vary where the abstraction layer is.

As a final note, the higher the abstraction layer, the harder it would be, I think, for an organic lifeform in a VR world to know they were in a VR world. Might make for a very plausible rationale in one of your stories (a lot more plausible than some guy on a couch with VR goggles on his head thinking he is actually in a VR world). Imagine a box into which you pour in a thousand rubber balls. Modeling the collision of all these balls is a lot of computing power. What if the VR simulation just tells your brain that you see a bunch of balls bouncing the way you expect balls to bounce? It is far less computing power to operate at that level of abstraction since it requires not much more than creating the equivalent of the previous sentence. It’s the difference between modeling the balls sufficiently that your brain would not object to what it sees and simply telling it, the balls bounce as you expect them to. And if you’re being told “you see consistent, expected input” how could you find the inconsistencies that might allow you to realize you are in a VR world?

The other type of VR: VR entities in a VR world (or VR entities interfacing with the actual world) is actually much simpler, to me. As wonderous as our brains are, there’s seems little reason why something similar cannot ultimately be modeled inorganically. Here you also can play with the level of sensory abstraction: maybe the VR entities interface to a retina-type visual input, maybe they only interface at the equivalent level of the human visual cortex that presents shapes. But you have additional advantages as an architect of such a VR world: no longer are you trying to interface with a free-running organic brain that might notice a pause. If your VR entities make something occur that requires a vast amount of computing power, you can just slow down time while you do the calculations. They will never notice the difference. You might be able to do that with a human brain to some extent (there does seem to be the equivalent of a clock in our brains, part of why time seems to slow in a crisis for some folks– not for me, though) but there are probably serious limitations on just how much you can slow things or how long you could pause a human brain. There are no such limitations in a VR world. You could pause it for millions of years while you built a more sophisticated computer and they wouldn’t know. Similarly, you could restart it from any point.

Lastly, why would there be VR worlds? I’ve already scoffed at The Matrix‘s rationale, which I think offends the sensibility of many in the engineering world. But there are sensible reasons for VR worlds, including ones where the participants might not realize that are in the world. Let’s  knock off the ones where the participants know they are in the VR world. These provide lots of interesting story hooks but are fairly simple, ranging from entertainment (who wouldn’t like better VR for their MMO worlds?) to prosthetics (maybe the seriously injured are offered a life in a VR world, possibly with excursions into the real world with a robotic presence) to the cyborg who interfaces to dangerous or military machinery and relaxes in a VR world. All of these provide opportunities for mistaken identity. It happens today when someone on-line is finds a man pretending to be a woman (and vice versa happens) but would a ‘brain in a jar’ even have a gender? Certainly, they would think of themselves as a particular gender but what if they were male, lost their body, and begin to think and act as a female? Kind of weird; might make an interesting story.

But what about a Matrix-like world where the person does not realize they are in a simulation? Here, it could be “deeper” entertainment. Perhaps, on an interstellar or intergalactic voyage where humans have made their bodies essentially immortal, they pass the time living VR lives from earth history or fictional worlds? Maybe some fraction of the crew is out of the VR at any one time, servicing the ship and creating the next VR sim. In such a setting, humans who were especially good at creating the VR worlds might be revered as the authors of entertainment meant to last millions of years. Or maybe the VR world is created for a scientist who wishes to study a period of history? It would not be actual history but perhaps he wants to set up Medieval type situation and see how real humans would act in those circumstances? What would it be like, for instance, if you were a medieval king raised from birth as a prince? Your VR world could tell you.

What about a complete VR world with VR entities? Here, there are plenty of reasons as well. Perhaps a scientist of the 33rd century is modeling early 21st century Earth and some of us are VR entities in this simulation (and many humans might be the equivalent of non-sentient drones that provide background). Perhaps this is getting a little uncomfortable for you: the pain in your life serves only to further some scientist’s doctoral thesis? Not every human is really a sentient being? This could be pretty awful. Might not be a pleasant place to be in but could make for some interesting story settings, though.

What about the Matrix premise that machines control the VR world? Rather than the idiotic “let’s use humans as batteries” why not have the machines using humans, either with organic brains, or actually completely VR modeled as a way for them to expand their consciousness. Maybe they provide a VR world so that they can figure out how to program themselves to experience more? Here, you might have a story where a ‘machine’ entity falls in love with an organic entity. The organic might never realize it but the machine entity might know that it is really something else entirely than what it’s lover thinks it is. Or maybe the VR simulation of organic life forms is the machine’s equivalent of reality TV: gives them something to chuckle about as they go on their alien lives.

One other trope of science fiction and fantasy is alternate worlds. I’ll probably post something on this before long but in the meantime, suffice it to say there might be real challenges moving between universes from the energy necessary to infectious diseases. It gets more complicated if your traveler has a different body in the other world. But if you were in a VR world, moving to another VR world would be fairly straight forward.

As always, I’m not trying to say you can’t use the standard trope where much of this sort of stuff isn’t worried about. Readers seem to accept it without much trouble. We engineers may roll our eyes at a particularly clumsy handling of VR but we like the thought experiment of alternate worlds as much as anyone else so we grin and bear it. But some of the real challenges of VR worlds might make for an interesting premise in your own science fiction stories. One person’s problem is another person’s opportunity, or in fiction, a way to make something new that hooks a lot of readers.


4 thoughts on “Virtual Reality and Worldbuilding

  1. Interesting, helpful post. I do a bit of sci-fi day-dreaming, especially lately, and you’ve given me some fun scenarios to mull over. Thanks for sharing!

  2. Pingback: World-building toolkit « amuteforamuse

  3. Pingback: Souls and Virtual Reality | M. Q. Allen

Comments are closed.