A prime goal of OBLIO is to be the ones our clients call when they want to do something new or unknown. In a perfect world, every project will find us wading into new technologies. You really can’t beat getting paid to learn - and this project gave us that in spades.
The Objective
The brief we were given for Bannister & Associates was fairly simple: can we get anywhere close to a digital recreation of the reminiscence experience from the film? In the film, people's realities have become so bleak that they have nothing to look forward to. The process of reminiscence lets them relive happier memories from their past via a machine.
From the beginning, we were aware of various machine learning and AI models that could bring static images to life. Using a custom driver video we could bring realistic motion to a user image. Deciding this was our best approach creatively, we teamed up with D-ID, a company that had partnered with My Heritage to do just that.
Side by side of a static image and the output from D-ID
After spending some time creating a proof of concept, we were confident that we could create an output that would be both exciting enough to users that they would share it, and fit perfectly within the narrative of the film.
Nostalgia never goes out of style. Truth is, nothing is more addictive than the past.
Nick Bannister (from Reminscence)
The User Experience
With the end result figured out, the next part was how do we get users there in an engaging and meaningful way. We wanted the experience to feel like the user was really showing up for their reminiscence. We built basic 3D representations of the exterior and interior of Bannister & Associates to set the scene.
To begin the experience, users upload an image (featuring a person) of the memory they want to relive. On the back end, the image is sent to D-ID for processing. While the image is processing, the user goes through a series of questions and interactions meant to elicit feelings of nostalgia and mystery. This serves two purposes. One, it helps set the stage for the end result. And two, there’s tons of processing that happens behind the scenes and this was a great way to hide that from the user.
The user goes through a series of questions and interactions meant to elicit feelings of nostalgia and mystery.
Conclusion
From the very first phone call, we could tell that this project was going to be a special one. When we sit and talk about what makes a great project, there are a lot of boxes that can get checked. Are the client’s goals met? Is it creatively fulfilling? Is it a meaningful user experience? Do we get to delve into areas that are cutting edge and exciting? There are many more, and over the years we’ve been able to check these boxes, but not always all the boxes on a single project. This project though? This is one we’d like to relive again and again!
Technologies
FRONT END – All of Oblio’s sites are built on a vanilla javascript framework that was initially created 5 or so years ago and has constantly been evolving since day 1, and for our WebGL rendering we, as always, used Threejs. Our 3d animation was created in Blender and imported as a GLTF while our 2d animation utilizes the GSAP library. That pretty much covers the tech on the front end, but behind the scenes there was a variety of things we did.
All of Oblio’s sites are built on a vanilla javascript framework that was initially created 5 or so years ago and has constantly been evolving since day 1.
BEHIND THE SCENES – Initially, after a user uploads an image to the site, we run it through Amazon Rekognition. This serves two functions, one to look for faces in the photo and two to screen for inappropriate content. If there isn’t a face in the image then the rest of our experience won’t work so we’ll ask the user for a new image. In the case where there is more than one face, we can use the information given to us to ask which of the detected faces the user was intending to use. Now that we have an image to animate we can send it over to D-ID. They made this step super easy with their API.
Once we receive the animation from D-ID we submit a render request to our render management API built by our friends at Sycamore Solutions. This is a scalable API that can manage assigning render tasks to a set of individual servers. If we are ever experiencing an influx of requests they could add additional servers to handle the extra stress.
Once the job was given to an individual render server it ran a script that we wrote to create the final video. There were 4 separate blender scenes that needed to be rendered and a bunch of ffmpeg calls to glue it all together. Optimizing this piece of the puzzle was an important step in making the experience smooth. Our initial renders took about 6 minutes per video, but we were able to optimize that down to 50 seconds.
End to end, the photo is passed through a number of time-consuming steps which are completely opaque to the user, and, in the end, a complex build with many layers appears simple and intuitive.
Company Info
OBLIO is a Digital Studio specializing in the use of technologies as design tools. We are combining and applying them in interesting ways for projects ranging from HTML, WebGL, VR, video, to even physical installations. Our award-winning team of inventors, artists, designers, and developers has worked on countless projects including over fifty major motion picture campaigns.