Neil’s Near Death Experience And The Future Of Movies

This is the written version of  a PowerPoint presentation I gave to a room full of digital media artists on December 1, 2012 at the Pixel Hub conference in Montreal.

I stand before you nearly fully unprepared for the coming revolution in motion pictures.

Actually, I’m unprepared in most ways but completely READY in another – but I’ll come back to that.

First, let me tell you why I’m UNprepared.

Making predictions about advances in art and, especially art that’s dependent on new technology, has a very checkered history. Even experts in a field get it spectacularly wrong.

Consider this prediction, for example: “The cinema is little more than a fad. It’s canned drama. What audiences really want to see is flesh and blood on the stage.”

Those sage words belong to Charlie Chaplin who said them in 1916 just a year or two before motion pictures began their 85 year run as the most popular dramatic art form in history.

So, I’m probably going to be wrong about what I predict for the future of movies. But I’m gonna do it anyway. Listen up.

Let’s start at the beginning. We know that movies must change.

Take a look at this chart.

You can see that revenues for motion pictures – in North America anyway – are flat for the last several years even if they’re increasing in Asia and elsewhere in the world.

Much more ominously, though, as shown in this chart, the number of tickets sold for movies in North America has been in decline for a decade.

Motion picture producers are responding to the decline of their industry by producing less product but spending more money on it.

The idea is to capture a bigger chunk of a declining market. The Avengers and The Dark Knight are good examples of successful attempts at that strategy. John Carter is a good example of a movie that missed.

Especially when an industry has been in decline for a decade, economic, technological and cultural forces will combine to remake that industry in some drastic way.

Sometimes this change spawns offshoots of an industry that develop into industries unto themselves.

Hope this photo works for you, Sheila Bali.

Ranching is a good example. The cost, revenues, scale and way it uses technology has changed the business of raising cattle dramatically in the last 50 years. There’s absolutely no economic need for horses in ranching anymore. All terrain vehicles take care of the horse’s job much better than horses ever could.

Nonetheless, the romance of the Old West lives on in rodeos, horse shows and lifestyle choices. There’s now a vast set of industries serving cowboy fantasists that includes horses & saddles, pickup trucks, fashion, guns, politics and music.

Some even more complex version of this evolution is happening to media today. So we can expect motion pictures to change too. But how?

Let’s examine some assumptions.

Let’s start with technology since nothing can change an industry as quickly and thoroughly as technology. We’ll consider those other industry changing forces along the way, too, but we’ll frame our investigation with technology.

It seems that pictures and sounds have been accompanying each other for a very long time. Apparently, the spot in a cave where you’re most likely to find cave paintings is also the spot at which the cave is most resonant for sound.

I’ll bet that our prehistoric forbears told each other stories about the fear that they felt when they stood before a powerful animal with a tiny, ineffective weapon but somehow found enough courage and physical strength to kill it for the benefit of their tribe.

For what are stories but ways for us to identify with someone who chooses to do something that’s worth doing and that she’s afraid to do and who to prevail must find emotional resources within that she didn’t know she had?

If the story is always some variation on this basic structure, then what’s the difference between stories?

Cave painting from the “Hall of Bulls” room in Lascaux, France.

Well…the personalities of the characters, the kinds of predicaments that the characters find themselves in, the era in which a story is set, the tone – comedic, dramatic, ironic, irreverent – but also the technology with which the tellers deliver the story.

I’ll bet that our prehistoric ancestors chose these resonant places in caves to tell their stories because the resonance enhanced the depth at which others could feel the storytellers’s fear and joy.

I’ll also bet, though, that a human being telling an audience a story while they looked at images of animals painted on a cave wall wasn’t the first time a storyteller used technology to enhance the emotional impact of a story.

I suspect strongly – if I had a time machine I’d say that I know for sure – that more than one of  our astute ancestors noticed how firelight flickered ominously on the faces gathered around the fire. I’m sure our ancestors used that  flickering to intensify the drama of their stories even before they discovered how to amplify the emotional impact of their voices.

That means that our forbears from the dimmest reaches of prehistory and we media-makers of today are on the very same path. Though we’re much further along that path today, we share with our ancestors the same intuitive drive to provide sight and sound experiences that have the greatest possible impact.

That idea has deep implications for how we think, what we think of others, how we want others to think of us…and how we entertain ourselves.

I bet that this is less nudity than you had in mind, Bryan Kochis.

Immersive entertainment may be the holy grail of entertainment because we instinctively want others to understand our thoughts and feelings at the deepest levels. In other words, the quest for creating immersive entertainment is also the quest to create the most perfect possible opportunity for empathy.

That aspect of human nature – the desire for others to understand our deepest feelings – is going to remain intact in human nature for a long time. Human nature has changed a lot in the last one million years but it’s scarcely changed at all in the last 10,000 years.

However, social and technological forces have created profound changes in the WAY that we pursue trying to be understood.

Our ability to immerse the senses of others has come a very long way indeed. Consider this video of a soldier with real combat experience immersed in an interactive game simulation. Listen to what he says when he comes out of the game.

BEFORE CONTINUING PLEASE PLAY THE VIDEO AT THIS LINK RIGHT HERE STARTING AT 13:32 and ENDING AT 16.56

I was impressed that a battle simulation could evoke in a professional soldier the same emotions that he experienced in actual combat.

Someday movies are going to be even more immersive than the Ultimate Battlefield sim we just saw. Someday we won’t be able to tell if what we perceive is physical reality or generated by some electronic means.

That’s a real human brain.

“Simulated Stimulation” is the novelist William Gibson’s name for a future technology that will stimulate in a person’s brain the full sensory experience of another. In the Cyberpunk universe of Gibson’s novels, “SimStim” has replaced television and movies.

And for any of you who just wondered whether the first SimStim experiences will be pornographic, the answer is yes. Now go to your room!

Someday we’ll have the technology to actually create SimStim and we won’t need movies or television anymore. But in between full on SimStim and the current state of movies, we’ve got a lot of steps. I’m going to speculate today about the fate of movies before they get all the way to Simulated Stimulation.

Of course, while the experience of going to a movie in the current day is a very long way from SimStim, movies HAVE adapted a lot of new technology. Going to the movies these days is way more exciting physically than it was in previous eras.

Think about what it’s like to watch the Transformers movies in 3D, for example, as compared to watching a very successful movie from the previous era, say, Dances With Wolves, for example.

But by making movies so much more physically exciting, we have changed what they are.

Digital animation and visual effects techniques, awesome advances in sound and stereo projection have enabled movies to evolve from being flat presentations of stories to being 3D experiences in themselves.

As much of a boon as it’s been, though, the advancement of movies technically is also a big part of the cause of the downward trajectory of motion picture attendance. We’ll address that in a moment.

Meanwhile, that first thought that popped into your head is correct. Motion picture attendance is not declining everywhere.

India’s movie audiences are already larger than North America’s.

The Chinese are building movie theaters at a furious rate. In the next several years, the number of tickets sold at Chinese theaters will surpass the number sold in North America.

But I submit that eventually the appetite to see motion pictures – in their current form at least – will begin to decline in India and China also as the cultures of those movie markets evolve and become more sophisticated.

So, why IS the number of tickets to see motion pictures declining in North America? What other, more satisfying, experiences are people spending their time and money on if they’re not going to the movies?

There are four answers to that question. And inside these four answers also lies the key to the future of motion pictures.

So, the short answer is:

1. video games

2. television (including spectator sports)

3. social media and the internet (including short form internet video, podcasts, internet radio, web sites, ebooks and so on) and…

4. on-demand video

There has been a lot of analysis of how these media have eroded movie audiences but, just to provide some context, I’ll mention that the video game industry generates revenue of about $67 billion worldwide. Worldwide theatrical motion picture box office, by contrast, was about half that in 2011– $33 billion.

Hit TV shows like Glee provide advertisers with 9 to 12 million viewers per show. Over 4 seasons that’s over 800 million viewers.

By contrast, a movie like one of the Transformers movies will deliver about 175 million unique viewers. That’s a marketing bonanza for sure but it’s way not as good as getting 10 million people to check in with your advertisers 22 times a year for four years.

Granted, many of a TV show’s viewers are the same people from week to week but what can be better than a consistent group of people whose buying habits you can influence?

Social media hasn’t figured out how to make money off of its users yet. It’s sure got numbers, though. Facebook and YouTube together have about 2 billion users. That’s a LOT of brain cells focused on social media.

The video on demand services – Netflix, Hulu and Amazon – together have about 30 million subscribers. That number is minuscule in comparison to the other encroachers on motion picture going but significant nonetheless…and growing.

What do each of these interlopers onto movie turf offer that movies don’t?

Video games offer interaction with the game and/or other players and, in many cases, an immersive environment.

Television has become what many of us had hoped that independent movies would become – a laboratory that enables its creative people to study character evolution at deep levels. TV also supports its creative people financially. Independent cinema doesn’t.

Social media offers users authentic interactions with other people on a non-real time basis and enables affinity groups to form without regard to geography. For social animals like us, that’s catnip.

On-demand video lets us watch what we want when we want.

Home theaters are still much less immersive than the best movie theaters but they’re good enough to keep a lot of people from going to the movies.

There are essential differences among all these media. Though these media compete with movies for the attention of audiences, they certainly don’t offer the same experience as seeing a movie.

Let’s consider the ideas in this video by Jim Banister. Jim is a co-founder and Chief Development Officer for say.so, a marketing company in Park City, Utah. Jim is also a former executive with Warner Online among many other accomplishments.

PLEASE WATCH JIM’S VIDEO, The Nature of Social Narrative, AT THIS LINK RIGHT HERE BEFORE CONTINUING TO READ THIS ARTICLE.

Jim did a great job of explaining the essential differences among the entertainment media that now compete with motion pictures.

In a nutshell, I think that the reason motion picture attendance is declining is that many movies are trying to provide an experience that other media – especially video games – are better at providing. I believe that games are becoming more complex and immersive and that movies are becoming less complex but more immersive.

Movies have become experiences that aim to provide physical sensation – which, like roller coaster rides, have intense but simple emotions attached to them – at the expense of more complicated emotions.

Audiences naturally choose the activity that provides the most satisfying emotional experience. In an increasingly large number of cases that more satisfying experience is Ultimate Battlefield 3 or Assassin’s Creed instead of the movie John Carter.

Nonetheless, I do believe that the future of storytelling via the medium of motion pictures has a bright future.

How could I possibly believe that?

1. First, all entertainment is an emotional rehearsal of one sort or another. The reason that we read novels, see plays, go to movies, experience Ultimate Battlefield simulations and so on is that we want to experience the emotions that placing ourselves in jeopardy produces in us. We want to see how well we handle the emotions that jeopardy elicits but we want to do it without placing ourselves in real danger.

2. Second, we’ve developed some very effective ways to experience and learn to control our emotions by creating highly realistic simulated environments that teach us to fly airplanes, respond appropriately in urban battlefield environments and so on.

3. Third, there are much deeper and more complex sets of emotions that we’d like to practice experiencing that flight simulators or battlefield simulators can’t provide.

But how do you get an audience to experience a protagonist’s thinking/feeling process at a deeper and more complex level than a flight or battle sim can provide?

First, the structure of the emotional experience which comprises:

– the order in which a character’s thoughts and feelings occur

–  the relative intensity of those feelings

– what triggers the feelings and

– the action and reactions that the feelings motivate

…is very delicate.

Thus, the emotions of this depth and complexity have to be imagined and created by people of extraordinary imaginative ability.

Second, and most important to little ole me because I risk Charlie Chaplin’s fate if I make as inaccurate a prediction about video games as he did about movies. Maybe in 50 years some smart ass will be speaking at a conference somewhere and will make fun of me by quoting my wildly inaccurate prediction about the future of video games. But I’m gonna do it anyway. I don’t think that the images and sounds you need to use to express complex thoughts and feelings can be canned. I don’t think video game sims will ever be able to provide emotional experiences at the complex end of the spectrum.

However, I submit that the only way to touch the emotions of viewers at a deep enough level is to immerse them as deeply in thoughts and feelings as the military guy was immersed in the physical sensations of the Ultimate Battlefield sim.

That is, we need to engage viewers with the imaginative reveries of our characters as their hearts guide them towards a decision or realization.

We attempted to do just that – and failed – on a picture called My Life. Why we failed, though, taught me a lot.

Bob Jones’s last roller coaster ride in “My Life”

I’ll tell the story of how we failed and what I learned a little later.

I submit that we need an even more compelling format than Imax 3D.

Imax 3D is way immersive but, to my way of thinking, not nearly immersive enough.

So what are the characteristics of a format that’s cooler than Imax-3D?

Well…

– It has to provide a level of immersion and interactivity to rival video game sims.

– It has to provide viewers with an understanding of character deep enough to exceed what television can provide.

– It has to provide social interaction at a level that competes effectively enough with social media and…

– it’s got to be visually and aurally compelling enough that people will leave their home theaters to experience it.

And here is where I, as I mentioned at the beginning of these remarks, feel completely prepared for the coming revolution in motion picture entertainment.

I’m here attending a conference about new media because it puts me in proximity of other people with the same interests so that we can share ideas. I like sharing ideas with other people and so do you. That’s why we’re all here.

We’re social animals. To explore ideas and emotions effectively we need to do it together. The movies of the future are going to be shared experiences where crowds gather to feel emotions together. I like feeling the audience vibe. Most other people do too. That’s actually the best reason to go to movies.

I liked the way that the military guy was really feeling the jeopardy in the Ultimate Battlefield sim. But he was in that sim alone.

Wouldn’t it be great if we viewers of motion pictures could get together and experience that deep a level of engagement with the thought processes of a protagonist in a motion picture?

Movies are trying hard to do that now and the result is what David Denby, in his book, Do The Movies Have a Future calls films that exhibit the “Conglomerate Aesthetic”. Denby doesn’t mean that as a compliment.

By “Conglomerate Aesthetic”, Denby means movies like the Transformers franchise. To create emotional experiences for their audiences, these sorts of movies rely on being able to elicit physical sensations – just like battlefield sims do.

Denby is right that mainstream movies currently provide shallow experiences. However, movies are currently at only one point along a constantly evolving trajectory. Mainstream movies aren’t always going to be stuck where they are now. Quite the contrary.

Motion picture distributors have learned that there is a market for movies that provide sensation but no lasting emotional or moral coherence. Movies like the Transformers franchise.

However successful movies made with the Conglomerate Aesthetic are, though, they won’t continue to be profitable for much longer. Eventually, no one will pay for a movie experience that provides only pure physical sensation mixed with a dash of identification with a protagonist because video game sims provide a better version of that experience.

That means that feature motion picture producers are going to have to learn how to provide movie audiences with emotional experiences that are different and more satisfying in some way than other media offer.

But no worries. Motion picture makers are a very creative group. As Edward J. Epstein points out in his book The Big Picture, Hollywood’s motion picture studios are nothing if not efficient learning machines. A motion picture distributor immediately assimilates the information that audience testing provides about one of their products and then immediately incorporates that new knowledge into the next product that the studio makes.

So, what kind of experience will the motion picture experience evolve to be?

I believe that motion picture makers will learn that providing compelling windows into the souls of others can be a profitable business.

To do this, though, we have to solve a very significant problem: the problem of the subjective point of view.

Movies now rely on showing what actors say and do for good reason. The subjective point of view most often shows viewers what a character is seeing and hearing through his or her eyes and ears – but in their physical reality. NOT in their imaginations. Evoking the contents of a character’s imagination directly with pictures and sounds is awkward and difficult. It often comes off as cornball.

Thus, the subjective point of view has most often been used for comedy. British TV’s comedy Peep Show is a good current example though it relies mostly on audio that reveals the inner dialogue of its protagonists.

There are some exceptions, of course. 2001:  A Space Odyssey did a passable job of creating a feeling of transcendence. I think The Diving Bell and the Butterfly used the literal, physical pov of its protagonist to compellingly evoke the feelings of being trapped and frustrated.

The Lord of the Rings powerfully evoked Golum’s inner struggle but did it by showing exterior images of his body while he vocalized his inner struggle. Of course, the character design of Golum contributed mightily to our ability to understand his personality and, in that sense, was a voyage inside his mind. However, as masterful a stroke of storytelling as Golum’s conflict with himself was, Peter Jackson and Fran Walsh still didn’t put viewers all the way inside Golum’s mind.

In my view, there’s really only one trajectory along which movies must develop. Really only one way to reach their potential of being special enough to lure viewers out of their houses and into theaters.

That trajectory is towards a fully immersive subjective pov – as close to SimStim as we can get with available technology and in a social setting.

My Life, the movie that I worked on and was disappointed about when we didn’t reach our goals for it, has a protagonist named Bob Jones. Bob, played by Michael Keaton, is a public relations executive. Bob’s wife is five months pregnant with their first child when Bob learns that he has only a few months to live. He seeks solace from a Chinese healer who induces him to look inward by laying on of hands. The Chinese healer’s laying on of hands puts Bob’s mind into a reverie that is both stimulating and troubling to him.

Our job at Sony Pictures Imageworks was to create imagery that showed the audience what Bob Jones experienced in his mind’s eye as he was being treated by the Chinese healer. We also created the visuals for what Bob experienced as he died at the end of the movie.

Fortunately, we had an animator, named Neil Eskuri, who had actually had a near death experience. As a teenager Neil had drowned and been revived with CPR.

Neil remembered vividly the visuals of his near-death experience. So we designed into the movie what Neil had actually seen when he almost died.

It was an incredible experience to work with Neil on this project. He shared with us what it actually looked and felt like for him to die. It was harrowing, profound and intimate all at once and one of the peak experiences of my life.

As moving as it was for me to work on this movie, though, I’m deeply disappointed with what went up on the screen. We reproduced faithfully what Neil saw when he almost died so we know that that part is right.

What’s not right is the way that audiences perceived what was on the screen.

We produced the picture in anamorphic format so it was on the high end of the range of picture quality for conventional projection. But it was still WAY short of adequately involving for the audience.

To make a movie that provides the impact that Neil’s near death sequence in My Life should have had, we needed a motion picture medium many times more compelling than the conventional flat format in which we produced and distributed that movie.

I predict that the motion picture entertainment format of the future is domed and that it will use high frame rate projection.

Of course Imax 3D has a domed version which is pretty compelling but, for my money, way not compelling enough.

Imagine if we had produced Bob Jones’s afterlife experience in My Life for presentation in a dome at a High Frame Rate.

What if we entirely filled the audience’s field of view with the image of the tunnel of white light down which Bob’s spirit flew? What if we had immersed the audience with the sounds that accompanied the images that Bob saw on his journey?

That would have been a much more powerful experience for the audience and maybe it would have made My LIfe more successful at the box office too.

This is where I think movies are headed. I think that movies will remain a passive experience – in contrast to video games – but will become way more immersive – in contrast to television and video on demand.

This is the sweet spot to which I believe movies will evolve. This is what will make movies special enough for audiences to leave their homes to see.

Feelings of this depth are the exact sorts of experiences that human beings thirst to understand.

Otherwise sober citizens of the United States fill churches every Sunday because people want to be reassured that there is life after death. They listen to their pastors tell stories about how death isn’t as final as it seems.

I’ve sat through quite a few of these pastors’s stories. Sometimes pastors are good storytellers but, generally, their presentations would be more engaging if they had more production value.

What if these churchgoers could experience at a deep – but still safe – level a faithful reproduction of someone’s actual experience of death and resurrection?

If church attendance is any indication, I think that there’s a big market for that.

Christ Church Cathedral in Southern California seats 2,746 worshipers.

There are a tremendous number of obstacles to overcome before dramatic feature motion pictures can be presented in domed theaters.

Many of these obstacles are technical even though we’ve got a good start at solving them. In fact, the very most technically advanced domed theater in the world is right here in Montreal – the Satosphere.

Exterior of Satoshpere in Montreal.

Another huge obstacle to overcome is the way that the grammar of film must change when presenting movies in domes.

In the dome movies that I’ve seen – most recently at the Gates Planetarium Theater courtesy of Dan Neafus the very astute manager of that theater – the pace of cutting, camera moves and internal motion dynamics of shots are all much slower than for movies shown in conventional theaters.

In fact, the entire editing grammar of close up, long shot, reverse shot etc. is out the window.

I’ve never seen even moderately paced cutting in a dome and I don’t wonder why. I think it would feel like my eyeballs were being sliced up if a shot lasted less than ten seconds in a dome.

Obviously, that’s a severe drawback when you’re trying to tell a story with temporal and moral coherence.

An Oscar winning director/producer named Ben Shedd has written extensively about the problems of cutting movies where the audience can’t see the edges of the motion picture screen.

Briefly, the problem is that when you can’t see the edges of the screen whatever motion is in the projected image feels like it’s happening to you. That’s because the horizon in the images projected onto a domed screen becomes the only reference point for balance when the image fills your entire field of view without edges – just like reality.

Thus, to keep audiences from becoming seasick, most dome movies have long duration shots with very subtle camera movement.

To solve this problem – and this idea seems obvious enough that it’s probably already been tried – what about using projection techniques to establish inside a domed screen a very large screen area but one with edges that the audience can see?

Motion picture makers could use the framing with edges that the audience can see to present information that requires conventional editing techniques. Maybe even most of the movie would be projected this way. Doing that would solve the vertigo problem yet still allow conventional editing.

Then, when the story called for maximum emotional impact – like the moments when the Chinese healer laid hands on Bob Jones and induced him to experience glimpses of the afterlife – the projected image could widen out so that it entirely filled the audience’s field of view.

Making the transition from a field of view with edges to a completely immersive point of view without visible frame edges presents immense creative problems. Yet, I’m sure that we can solve them. That’s what we do as motion picture makers.

If any of you in the audience know of experiments along the lines of what I’m proposing, I’d be very interested to hear about them.

I’m also very interested to see how high frame rates are going to change and, I hope, improve the dome viewing experience. Two other speakers, Tim McGovern and Tim Huber, here at PIxel Hub will provide some perspective on that issue.

My suspicion is that high frame rate production and exhibition can provide at least one of the pieces of the puzzle that will make movies in domes both technically feasible and compelling.

No one has seen 120 frames per second at 4K resolution yet but my pal Tim Huber has seen 120 fps at 2K. He says that that image is amazingly, though not completely, 3D.

Maybe 120 fps at 4K will provide enough 3D that we can dispense with the glasses and the general lack of wonderfulness of stereo projection.

Maybe we’ll still need stereo projection but, if so, the quality of the 3D is sure to be way better than it is now.

High Frame Rate projection seems like it would make the editing problem even worse, though, so we’ll need some pretty smart guys – maybe some of you guys who are sitting in this room – to help us solve this problem.

However, I do believe that even greater immersiveness is the direction that movies will take…and that we’ll solve the problems that we need to solve to make deeply immersive movies the compelling glimpses of another’s soul that our pre-historic ancestors yearned for and that we yearn for still.

Thank you.

George Merkert