Stephen Rosenbaum was tasked with digitally creating one of the world’s most famous performers, Michael Jackson, to promote a new song, Slave To The Rhythm, at the live event of the Billboard Awards. It was a digital performance with excellent facial animation built on a range of technologies from LightStage facial capture to motion capture.
Given the enormous importance of Michael Jackson to music generally and his global reputation for stunning live performances, the job of allowing an audience to once again see him perform brought with it a justifiable amount of pressure. Clearly, after decades of enjoying his performances, the audience knows how Jackson moved, they know how he looked, and literally anything less than stunningly accurate wasn’t going to work.
Stephen Rosenbaum completed the project from his base in San Francisco. In fact, as fxguide spoke to Rosenbaum he was sitting much where he was when he started his career when it was ILM – an irony that was not lost on the visual effects supervisor. “This is the room where I started my career 25 years ago, so I am literally sitting back in one of my old offices where Joe Letteri sat, where Scott Anderson sat, where a bunch of us started back in the early days of ILM. And the decor has not changed,” he joked.
Click to see a high res version
Prior this project, Rosenbaum and his team were heavily involved in modernizing a creature pipeline. It was this wealth of experience that made them a great choice for this production. The Michael Jackson Estate brought the project to Rosenbaum and Pulse Evolution (A specialist human animation company) in October of 2013. In light of the current migration of projects from California to Canada and beyond, Rosenbaum was particularly pleased to be doing such high quality work, but in a new arena and expanding the massive skill base of the artists still in California. “One of the things that is so exciting about this project was that we are applying our skills and expertise in an area that is largely untapped,” he says.
To help with accuracy in dancing, styling and approach, Rosenbaum was offered the chance to work with dancers, performers and wardrobe specialists who had all worked personally with Michael Jackson. This proved invaluable. In addition to a wealth of reference footage, Rosenbaum could talk directly to people who could speak to how Michael Jackson approached his craft and how he creatively solved problems. “I could turn to them and say here is a specific piece of performance, I was going to have him do this, and they would be able to say, ‘Well, actually Michael might have done this – turned his head to the side instead of back – because this was the way Michael would be thinking,’” explains Rosenbaum.
Rebuilding a legend
The decision of what age Jackson should be in the performance was, at least in theory, wide open. The performer could have been almost any age, as his career spanned decades. In reality, an actual ‘plaster’ life cast of the singer’s face made in 1997 meant that this would be the age of the Jackson the world would see: 39. It was at this age that the veteran special effects expert Stan Winston made a life cast of the singer for the project Ghosts. The team got access to that original life cast and had the mold scanned. As great as Stan Winston’s work always was, the very weight of the material and the agents used to avoid it sticking to a subject’s face can fill in skin pores. So while the life cast was a great measure of the performer’s dimensions it was not as accurate as today’s techniques for accurate skin pore texture. “It got us only about 75% of the way there,” estimates Rosenbaum.
“The pressure from the alginate (plaster) not only pushed out some of the pore features but some of the facial features, chin, general skin,” adds Rosenbaum. “Michael has a very unique face, and part of his face was softer (thus it was more affected), so we had a bit of a challenge, we also had a physiological challenge of what we think Michael Jackson looked like at age 39, what we perceived he looked like, but from the hundreds of images we got from the Estate, we arrived at what he should actually look like.”
The other issue is that a life cast is just one neutral pose. In modern terms, if an actor is being scanned it would be normal to get multiple FACs poses in addition to the neutral pose, so that animators could see how the face looks in various poses. With just the one life cast the team was limited.
The combined issues of not having fine skin pore detail and no additional poses lead Rosenbaum to decide to get some additional reference in to provide extra detail. What would be important in the final is providing organic micro displacement maps to add to a traditional meso level bump that one might get from a cast. Interestingly this was most visible in creating a specular breakup and it was this that greatly improved realism. The team found various additional reference which could provide the missing texture details and provide reference for how a face moves between key poses. While these people were not, of course, the real Michael Jackson, the team only needed to extend in high frequency detail missing from the real life cast and get controlled reference footage. While a lot of real footage of Michael Jackson was used as reference, FACs poses are very particular and recorded under control lighting. The team also got additional key iconic expressions filmed to analyze and study. “Michael had a very distinct way of emoting,” says Rosenbaum, “so we got them to try and match some of the iconic expressions from his hero performances, and then on top of the FACs expressions do additional key concept expressions, and that was central to our success – not just relying on what the facial muscles technically do, but also how Michael might have emoted.”
Unlike some earlier attempts at this performer re-creation, Michael Jackson’s mouth was very visible on stage. He performed without using a hand held mic held in front of his mouth as was the style of Tupac Shakur. This meant the mouth or lip sync needed to be much more accurate than previous staged ‘holograms’. Michael Jackson had very distinctive hair that needed to be recreated. “Unlike Tupac who was bald, Michael deliberately used his hair during a performance when he is dancing,” notes Rosenbaum. “He would use it as part of his expressive performance.”
Key academic research has been done recently in the area of micro-geometry for improved facial accuracy. Rosenbaum had the desire to use this new innovation in providing micro geometry displacement. These maps “proved to be hugely valuable to us and a significant step up in look development than what we’d done on prior projects,” he says. The logic of micro-geometry is very interesting. It would be reasonable to think at first glance that any character not seen in extreme closeup would not benefit from very small scale micro geometry. In fact it could be seen as adding to render times, disk space and something to easily be dismissed. But what has been found by the recent research is that while the actual pore detail is itself not read, the much larger specular highlight that a cheek or forehead may exhibit does change rather remarkably with the addition of the micro geometry of the skin of the face. While the skin pore resolution is very small, the highlight may be a quarter of a face in height in any one frame, with the textural quality and – more importantly – the realism, is greatly enhanced by modeling the skin to seemingly incredible levels of detail.
“What we found is that the specular contribution to skin is vital,” indicates Rosenbaum. Previous research had suggested two specular lobes contributing to the overall response on facial skin, “and we in fact incorporated three,” says Rosenbaum. “You don’t, at first glance, realize that you are looking at three levels of specular contribution. What you see is subtle grading and fall-off across different surfaces such as the cheek or the forehead or chin – they all have different responses within the highlight itself. It is actually the key contributing factor we found to realistic looking skin.”
Highlights seen across micro geometry, while being more realistic, can actually be more intense – in a way that a make up artist might want to reduce if seen on a real set. For this reason, and for the realism of what a performer such as Jackson would actually wear, ‘digital’ makeup was added to mimic the correct look that the singer might actually have used. This was a key part in moving from general realistic skin to a realistic skin on Michael Jackson, performing at a major event. This digital makeup was based on “a lot of tests we did with real makeup,” says Rosenbaum. The team examined someone with and without makeup to note the differences in specular contribution that professional makeup would contribute and the final solution actually involved multiple layers of this subtle but important ‘digital’ makeup.
The final maps were 6 x 8K, and rendered in RMS 18 (RenderMan). “Everything was ray traced,” Rosenbaum commented. While the team used high end SubSurface Scattering (SSS), “what was giving the most realism was the micro-geometry.” Additional R&D or work in SSS would not have made much difference, especially given the performer’s makeup which naturally reduces the effect of SSS, according to Rosenbaum. The face did have a varied SSS across the face, but for the lips, which are normally central to any SSS discussion, the team still found that it was the micro displacement based on the new research work that lead to the greatest return in realism. “The fine amount of displacement that was being revealed by the specular contribution,” explains Rosenbaum, “you get this additional octave of specular response – that is what made them realistic – versus say varying amounts of sub-surface contribution.”
For the singer’s eyes, the team got great help from the Estate and the dancers and performers that had worked with Jackson. “He always commanded the crowd,” says Rosenbaum. “What we found is that someone may dance for themselves and where they might be going, but Michael was such a consummate performer, he was less thinking his performance than feeling his performance. He would look out and make eye contact deep into the audience.”
“While the face was where we emphasized critical look development efforts, a lot of time and energy was put in to the overall body and character performance to reinforce accurate Michael likeness,” commented Stephen Rosenbaum.
Costume
For the great skill as a singer and dancer, Michael Jackson was also a showman and his costumes were always important. Here the team were lucky enough to able to work with Michael Jackson’s costume designer, Michael Bush. Not only was this helpful in planning, but the designer was accessible right through production to advise. Bush had created most of MJ’s personal and concert tour wardrobe in the past. The costume seen on the stage is a new design of an outfit that might have been worn during this period of the singer’s career, he created this original piece just for the show. “We had a lot of conversations about how Michael wore his clothes and the types of clothes he wore and how they fitted on his body,” says Rosenbaum.
The new work had an eye to the future. In many respects just as the life cast became invaluable years later, the team wanted to make sure this work would be well documented in case this was wanted or needed for any future project. ”We went in building the highest quality assets,” notes Rosenbaum. “We built in facial detail, the hair, the eyes – everything to the highest quality. In fact, we were working at 7.5K resolution,” a resolution significantly higher than was projected on the night.
“We built the asset to be re-used,” adds Rosenbaum. “I’ve always been a fan, and I think he looked great.” The results were well received. After the performance the team showed it to Jackie Jackson, Michael’s brother. “We replayed it to him after the show, and he was smiling throughout and he kept saying ‘That’s Michael, that’s Michael’ – you could tell he believed what we had done was true to Michael’s persona and his style of performance.”
SOURCE: FX
